Sunday, May 10, 2015

The application of evidence based approaches 1 - Evidence Based Medicine

The discussion on Monday Forum - more on evidence based approaches looked at some of the issues associated with evidence based approaches. One of the issues from my viewpoint in that discussion lay in the need to distinguish between the importance of evidence and the application in practice of what are called evidence based approaches whether in medicine, management or public policy. I thought therefore that it might be helpful, at least to me, to consolidate some previous writing on the topic, starting with evidence based medicine.

By way of background to what follows, one of my major interests over the last quarter of 2006 and the first quarter of 2007 lay in the development of what I called a discipline of practice, a discipline focused on the way professionals practice their profession. This drew me into a discussion of evidence based approaches, starting with evidence based medicine.

The material that follows in this post consolidates the three posts on evidence based medicine that I wrote at the time. For time reasons, I haven’t attempted a complete edit, just some simplification. In later posts, I will look at the application in management and public policy.

INTRODUCING EVIDENCE BASED MEDICINE

Evidence-based medicine (EBM) is an attempt to more uniformly apply the standards of evidence gained from the scientific method, to certain aspects of medical practice. Specifically, EBM seeks to apply judgements about the inductive quality of evidence, to those aspects of medicine which depend on rational assessments of risks and benefits of treatments (including lack of treatment). According to the Centre for Evidence-Based Medicine, "Evidence-based medicine is the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients."[1] Cited from Wikpedia.

In this post I want to continue my discussion on the development of a discipline of practice, a discipline focused on the way professionals practice their profession, by introducing in a preliminary way the case of evidence based medicine.

I first came across the concept in 1998 when I started as CEO of the Royal Australian (now Australian and New Zealand) College of Ophthalmologists. My instinctive first reaction was to say that's odd, I thought that all medicine was evidence based. In fact, that's far from true.

To understand this, we need to look at the way in which doctors are trained, as well as the way in which new medical approaches develop.

As with all professions, training starts with the previous body of knowledge relevant to that practice. For practical reasons, much of this has to be taken for granted. The trainee professional simply has to learn those elements required to begin practice.

The trainee then has to learn to apply that knowledge in practice. In the case of medicine this is done especially in hospitals working under the supervision of a qualified doctor who passes his/her knowledge and experience onto the trainee while checking their application. Again, the trainee is acquiring the current wisdom.

These broad processes continue throughout a professional training that can, in the case of medical specialists, extend as long twelve or thirteen years.

Once the doctor begins practice, he/she continues to learn from experience with patients. Doctors also learn through contact with other doctors and are expected to maintain a program of continuing professional education to keep them in touch with latest developments. Similar approaches and programs apply in other professional areas.

None of this will seem strange to any professional from any discipline. Yet in the case of medicine the process has proved to be seriously flawed. Evidence based medicine attempts to address these flaws.

OVERVIEW OF EVIDENCE BASED MEDICINE

Evidence-based medicine (EBM) is the integration of best research evidence with clinical expertise and patient values ... When these three elements are integrated, clinicians and patients form a diagnostic and therapeutic alliance which optimizes clinical outcomes and quality of life. Centre for Evidence Based Medicine

While this quote focuses on medicine, it also captures four key elements relevant to all professions and professionals.

The first element is research evidence, essentially what works and why.

The second element is professional expertise, our capacity to understand and apply our professional knowledge in the circumstances of the particular case.

The third element is the patient or client, each with their specific attitudes and needs.

The final element is the integration of the first three elements - the diagnostic and therapeutic alliance - to provide the solution that best meets client needs.

Before continuing my analysis I should note that the Australian Broadcasting Corporation's Radio National has just completed a rather good two part series - Facing the Evidence - on evidence based medicine. The first part is available here in transcript, the second here.

Returning to my theme, when you look as I did in my last post at the standard way all professionals are trained, you can see that that training focuses on the transfer of existing knowledge and skills to the new professional, knowledge and skills that that professional then applies. The professional then builds on this base through practice, at the same time using various professional development activities to try to keep in touch with new developments.

But what happens if that existing knowledge base is in fact wrong? How might this arise? To quote from the first part of the ABC program:

Every day doctors and other health professionals use treatments that are harmful, or fail to use therapies that have been proven to work. In the US there is so much medical error that Congress has directed the Institute of Medicine to develop a strategy to improve the quality of care. In its initial report the Institute noted perhaps as many as 100,000 Americans die every year from medical errors, including the use of inappropriate treatments. That's much more than from car accidents, breast cancer or AIDS. Many more suffer side effects and unnecessary costs

How could this happen?

Part of the problem here lies simply in the placebo effect, that fact that patients respond just to the fact of treatment. So the treatment appears to work, thus supporting the original judgment.

Part of the problem also lies in the fact that individual outcomes can be affected by so many variables and over a considerable time horizon so that the fact of adverse outcomes may not be clear in an individual case or, if clear, may be due to a whole variety of factors external to the treatment itself. There is a linked issue here that relates to the size of the population.

Given that individual outcomes vary greatly, the fact that there is a problem and its scale and scope may only become clear if you look at a population as a whole. That is, the individual professional may have no easy way of detecting the problem in his/her individual cases.

A further problem lies in the nature of the models used.

All professions use models to try to explain a complex world. In economics, for example, models are a common method used to analyse economic behaviour and to suggest possible responses at firm and public policy levels. In the case of medicine, biological models are common.

The problem with all models is that they involve selection of key variables and the specification of relations between those variables. Get either wrong, and outcomes may be very different from those projected by the model.

Doctors have always been concerned about adverse or unexpected outcomes.

In 1972 Professor Archie Cochrane, a Scottish epidemiologist published Effectiveness and Efficiency: Random Reflections on Health Services. This plus Cochrane's subsequent advocacy caused increasing acceptance of the concepts behind evidence-based practice. Cochrane's work was honoured through the naming of centres of evidence-based medical research — Cochrane Centres — and an international organisation, the Cochrane Collaboration.

The explicit methodologies used to determine "best evidence" were then largely established by the McMaster University research group led by David Sackett and Gordon Guyatt. According to the Wikipedia article on evidence based medicine, the term "evidence-based medicine" itself first appeared in the medical literature in 1992 in a paper by Guyatt etal. (Guyatt G, Cairns J, Churchill D, et al. [‘Evidence-Based Medicine Working Group’] "Evidence-based medicine. A new approach to teaching the practice of medicine." JAMA1992;268:2420-5. PMID 1404801)

From this point, the spread of the concept and its subsequent inclusion in professional training was rapid.

As with any other approach, evidence based medicine has its own methodological problems. However, it also has important lessons for other fields of professional practice.

PROBLEMS WITH EVIDENCE BASED MEDICINE

In my last post on evidence based medicine, I suggested that professional training focused on the transfer of existing knowledge and skills to the new professional, knowledge and skills that the professional then applied. The professional subsequently built on this base through practice, at the same time using various professional development activities to try to keep in touch with new developments.

I then posed the question what happens if that existing knowledge base is in fact wrong, looking briefly at the reasons why this proved to be the case in medicine, a discovery that had led to the development of evidence based medicine. I concluded that, as with any other approach, evidence based medicine had its own methodological problems. However, it also had important lessons for other fields of professional practice.

The Quality Movement and Quantification

In a series of posts on my personal blog, I explored some of the changes that had taken place in public administration since the war, looking at the influences on those changes.

In one of those posts I looked in part at the way in which standards, the Quality Movement and the importance of measurement had become major global influences. I also suggested that the outcomes here had not always been positive.

Evidence based medicine forms part of the global standards and quantification revolution and suffers from some of the same weaknesses. These weaknesses need to be recognised.

Problems with Evidence Based Medicine: Perception Bias

The first problem can be called simply perception bias.

Evidence based medicine is neither value nor perception free. The questions selected for test and evaluation, a process that can be very expensive, are influenced by prevailing views. Valuable alternatives may be excluded simply because they fall outside conventional wisdom. As evidence based medicine becomes the dominant mode, the effect may, as it has been in other areas, to actually narrow fields of investigation and action.

This links to a second problem, one that I have discussed before, the tendency for all professions to see answers within a frame or world view set by their profession.

A lawyer will give you a legal answer to a problem, a doctor a medical answer. If you have a back problem and see a surgeon, he/she is likely to think about surgical solutions. Go to a chiropractor with the same problem and he/she is likely to recommend spinal manipulation. So professional background helps determine the way the problem is defined, the solution applied.

This flows through into the application of evidence based approaches because the things tested are generally set within the frame of the tester. So evidence based medicine focuses on the efficacy of medical treatment and may leave non-medical alternatives aside.

Problems with Evidence Based Medicine: Causation

As part of my history honours year in my first degree I studied philosophy of history under Ted Tapp. Ted was a reflective man who required us to think about, to debate, the philosophical underpinnings of science and scientific method.

One core problem was the difference between correlation (a and b) as compared to causation (if a then b).

This problem applies in evidence based approaches. Just because a study shows an apparently strong relationship between a treatment and positive patient outcomes does not necessarily say anything about the causal relationship between the two. This has to be deduced and further tested.

Problems with Evidence Based Medicine: Problems of Epidemiological Studies

By its nature, evidence based medicine deals with large groups, populations.

As trials become larger and more complex, it becomes more difficult in statistical terms to establish significant relationships, to separate the effects of different variables. This creates another problem, the establishment of a clear relationship between the outcomes of trials at population level and subsequent application at individual level. As the Wikipedia article notes:

Critics of EBM say lack of evidence and lack of benefit are not the same, and that the more data are pooled and aggregated, the more difficult it is to compare the patients in the studies with the patient in front of the doctor — that is, EBM applies to populations, not necessarily to individuals.

This can create very real difficulties for individual clinicians, leading Tonelli to argue in The limits of evidence-based medicine that:the knowledge gained from clinical research does not directly answer the primary clinical question of what is best for the patient at hand.

Tonelli concludes that proponents of evidence-based medicine discount the value of clinical experience.

Problems with Evidence Based Medicine: Impact of the Observer

Another problem with evidence based medicine, one often seen in all evidence based approaches, is the way the observer affects the observed. This happens at several levels.

The first problem is that the simple act of participation in the trial may have some and not clearly seen impact on individual outcomes. In medicine, this is usually managed by use of a control group using a placebo. The efficacy of the treatment is then measured by the difference in outcomes between the control group and those receiving the treatment.

A second linked problem is the impact on patient behaviour of the trial itself. By their nature, clinical trials are closely managed. This means that patient compliance with the treatment routine is likely to be high.

This need not hold in subsequent clinical use since ordinary patients are more likely to fail to follow treatment processes by, for example, failing to take medication exactly as prescribed. This means that actual patient outcomes may not be as good as the trial results.

Problems with Evidence Based Medicine: Limitations in Application

A further problem is that the most rigorous gold standard approaches dictated by evidence based medicine can only be applied in narrowly defined circumstances, leaving a range of medical approaches that have to be tested by less rigorous means.

This should not matter so long as the limitations are recognised. In practice, it risks introducing two distinct distortions into the medicine and the health system. The first is the risk that investigation may be biased towards those things that can be measured through more rigorous techniques, reducing thought and investigation in areas less amenable to measurement. The second related risk is that treatment itself may become biased.

At clinician level, this links back to my earlier point about perception bias. Doctors trained in evidence based medicine may, consciously or unconsciously, come to focus in treatment terms on those things that can be measured, ruling out other less easily measured options.

This tendency may be reinforced by actions from those managing or funding the provision of health care services who may refuse to allow/pay for certain types of services notwithstanding the views of individual clinicians.

Conclusion

These problems do not detract from the potential value of evidence based medicine, but they do illustrate the need for care in application or the cure may be worse than the disease.

12 comments:

Winton Bates said...

It was interesting to read this just after having read an article in the Sun-Herald about restricting surgery for various categories of patients including smokers and obese people. There does seem to be some recent evidence on risks of survival to support this approach, but I suspect that some surgeons have adopted it over a long period.

On a slightly related matter, I don't think your article mentions cost. Given the extensive use of public money in medicine it is hard to see how evidence-based approaches can avoid the somewhat crass issue of how to get the best bang for the bucks available. That also raises the question of how to measure the "bang", presumably taking into account quality of life as well as extension of life.

Anonymous said...

Jim

You are not suggesting this is a unique insight? i.e. what you set out seems perfectly reasonable, and even predictable.

Anyway, 'jazz hands' for explaining the processes and potential problems in language I can understand.

kvd

Jim Belshaw said...

No, not a unique insight, kvd. However, the pattern recurs wherever evidence based approaches are applied or, perhaps, misapplied. That was my point in regard to DG's comment.

Note the date by the way when the term evidence based medicine was apparently first used, 1992. The particular model is quite recent. It appears to have emerged in several areas around the same times and is a direct outgrowth from the quality and quantification movements that began in the 1950s.

More in a moment.

Jim Belshaw said...

Some surgeons have, Winton, but on value judgement rather than medical grounds.

On the cost issue, evidence based medicine is concerned first with improved professional practice, doing what we do better. Cost is obviously an issue with new technology and to patients and governments who have to pay. This creates a rationing effect, doing things cheaper rather than better or not doing them at all.

Evidence based medicine might conclude, for example, that a service that a government is paying for, or a patient, yields very little benefit. Or it might conclude that a service or procedure would yield a great benefit but at a cost. Then patient or government has to decide whether it can afford that cost.

Anonymous said...

The Commonwealth pays for a multitude of services through its health budget for which there isn't the slightest scintilla of an evidence base as to their effectiveness, let alone a net social benefit or even a cost effectiveness test - e.g. a large chunk of GP services (for which there is a guaranteed benefit entitlement under the Health Insurance Act, regardless of their content); much diagnostic imaging (e.g. static two-dimensional back x-rays and standalone plain abdominals); and most so-called 'cognitive' services delivered by community pharmacists (remunerated under the outrageous Government-Guild Community Pharmacy Agreements). Surgery is a long story, but epitomised by hysterectomy.

DG

Jim Belshaw said...

Which GP services DG? A large chunk is a big claim.

Anonymous said...

Most GP services are time-, rather content-based. Other than their frequency and intensity, they are hence difficult to audit. GPs are free to introduce whichever 'style' of practice appeals. GPs practising alternative medicine, new age therapies and homeopathy are now very common - to the extent that a former President of the AMA (Kerryn Phelps) is one of their leading exponents. One of the best ways of maximising the likelihood of quality GP primary care is patient cost sharing - to make patients more conscious of their true needs and the reputation of the services they purchase.

DG

Jim Belshaw said...

DG, isn't that two hypothesis, one about the problem and the other the solution?

Jim Belshaw said...

Hi again, DG. Thinking further about this, on your price/choice point, there are parts of Australia where bulk billing is absent. Are you aware of any studies that compare doctor/patient behaviour between areas where bulk billing is common and where it is not?

Anonymous said...

Jim, a quite a deal on this - see for example, the landmark Rand Health Insurance experiment (written up in a multitude of publications). The evidence is that cost sharing encourages patients to make more rational decisions about their true needs and to choose doctors on the basis of their skill and reputations. This also keeps doctors up to the mark. For argument about the effect of greater competition on doctor / patient behaviour see: http://cis.org.au/publications/research-reports/article/5521-towards-a-more-competitive-medicare-the-case-for-deregulating-medical-fees-and-co-payments-in-australia

DG

Jim Belshaw said...

Hi DG. Have read the US stuff. Have downloaded your paper to read.

Anonymous said...

This would concentrate the minds of health consumers on what doctors charge instead of what government pays, and engineer a shift towards greater competiveness in setting medical fees

"Health consumers"? That would be sick people, I guess.

I dunno, but despite my general agreement with all of DG's comments in this thread thus far, I find it very hard to see myself (or wife/son/daughter) as a "health consumer" as such. More like unwilling participants in a badly regulated lottery of expertise licensed by government; protected by the AMA.

It disturbs that any concept of a "level playing field" or "all else being equal" is or can be applied to a class of persons (sick) and the professionals (doctors) servicing same. Anyway, that's just imo.

kvd