The application of evidence based approaches 1 - Evidence Based Medicine

The discussion on Monday Forum - more on evidence based approaches looked at some of the issues associated with evidence based approaches. One of the issues from my viewpoint in that discussion lay in the need to distinguish between the importance of evidence and the application in practice of what are called evidence based approaches whether in medicine, management or public policy. I thought therefore that it might be helpful, at least to me, to consolidate some previous writing on the topic, starting with evidence based medicine.

By way of background to what follows, one of my major interests over the last quarter of 2006 and the first quarter of 2007 lay in the development of what I called a discipline of practice, a discipline focused on the way professionals practice their profession. This drew me into a discussion of evidence based approaches, starting with evidence based medicine.

The material that follows in this post consolidates the three posts on evidence based medicine that I wrote at the time. For time reasons, I haven’t attempted a complete edit, just some simplification. In later posts, I will look at the application in management and public policy.

INTRODUCING EVIDENCE BASED MEDICINE

Evidence-based medicine (EBM) is an attempt to more uniformly apply the standards of evidence gained from the scientific method, to certain aspects of medical practice. Specifically, EBM seeks to apply judgements about the inductive quality of evidence, to those aspects of medicine which depend on rational assessments of risks and benefits of treatments (including lack of treatment). According to the Centre for Evidence-Based Medicine, "Evidence-based medicine is the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients."[1] Cited from Wikpedia.

In this post I want to continue my discussion on the development of a discipline of practice, a discipline focused on the way professionals practice their profession, by introducing in a preliminary way the case of evidence based medicine.

I first came across the concept in 1998 when I started as CEO of the Royal Australian (now Australian and New Zealand) College of Ophthalmologists. My instinctive first reaction was to say that's odd, I thought that all medicine was evidence based. In fact, that's far from true.

To understand this, we need to look at the way in which doctors are trained, as well as the way in which new medical approaches develop.

As with all professions, training starts with the previous body of knowledge relevant to that practice. For practical reasons, much of this has to be taken for granted. The trainee professional simply has to learn those elements required to begin practice.

The trainee then has to learn to apply that knowledge in practice. In the case of medicine this is done especially in hospitals working under the supervision of a qualified doctor who passes his/her knowledge and experience onto the trainee while checking their application. Again, the trainee is acquiring the current wisdom.

These broad processes continue throughout a professional training that can, in the case of medical specialists, extend as long twelve or thirteen years.

Once the doctor begins practice, he/she continues to learn from experience with patients. Doctors also learn through contact with other doctors and are expected to maintain a program of continuing professional education to keep them in touch with latest developments. Similar approaches and programs apply in other professional areas.

None of this will seem strange to any professional from any discipline. Yet in the case of medicine the process has proved to be seriously flawed. Evidence based medicine attempts to address these flaws.

OVERVIEW OF EVIDENCE BASED MEDICINE

Evidence-based medicine (EBM) is the integration of best research evidence with clinical expertise and patient values ... When these three elements are integrated, clinicians and patients form a diagnostic and therapeutic alliance which optimizes clinical outcomes and quality of life. Centre for Evidence Based Medicine

While this quote focuses on medicine, it also captures four key elements relevant to all professions and professionals.

The first element is research evidence, essentially what works and why.

The second element is professional expertise, our capacity to understand and apply our professional knowledge in the circumstances of the particular case.

The third element is the patient or client, each with their specific attitudes and needs.

The final element is the integration of the first three elements - the diagnostic and therapeutic alliance - to provide the solution that best meets client needs.

Before continuing my analysis I should note that the Australian Broadcasting Corporation's Radio National has just completed a rather good two part series - Facing the Evidence - on evidence based medicine. The first part is available here in transcript, the second here.

Returning to my theme, when you look as I did in my last post at the standard way all professionals are trained, you can see that that training focuses on the transfer of existing knowledge and skills to the new professional, knowledge and skills that that professional then applies. The professional then builds on this base through practice, at the same time using various professional development activities to try to keep in touch with new developments.

But what happens if that existing knowledge base is in fact wrong? How might this arise? To quote from the first part of the ABC program:

Every day doctors and other health professionals use treatments that are harmful, or fail to use therapies that have been proven to work. In the US there is so much medical error that Congress has directed the Institute of Medicine to develop a strategy to improve the quality of care. In its initial report the Institute noted perhaps as many as 100,000 Americans die every year from medical errors, including the use of inappropriate treatments. That's much more than from car accidents, breast cancer or AIDS. Many more suffer side effects and unnecessary costs

How could this happen?

Part of the problem here lies simply in the placebo effect, that fact that patients respond just to the fact of treatment. So the treatment appears to work, thus supporting the original judgment.

Part of the problem also lies in the fact that individual outcomes can be affected by so many variables and over a considerable time horizon so that the fact of adverse outcomes may not be clear in an individual case or, if clear, may be due to a whole variety of factors external to the treatment itself. There is a linked issue here that relates to the size of the population.

Given that individual outcomes vary greatly, the fact that there is a problem and its scale and scope may only become clear if you look at a population as a whole. That is, the individual professional may have no easy way of detecting the problem in his/her individual cases.

A further problem lies in the nature of the models used.

All professions use models to try to explain a complex world. In economics, for example, models are a common method used to analyse economic behaviour and to suggest possible responses at firm and public policy levels. In the case of medicine, biological models are common.

The problem with all models is that they involve selection of key variables and the specification of relations between those variables. Get either wrong, and outcomes may be very different from those projected by the model.

Doctors have always been concerned about adverse or unexpected outcomes.

In 1972 Professor Archie Cochrane, a Scottish epidemiologist published Effectiveness and Efficiency: Random Reflections on Health Services. This plus Cochrane's subsequent advocacy caused increasing acceptance of the concepts behind evidence-based practice. Cochrane's work was honoured through the naming of centres of evidence-based medical research — Cochrane Centres — and an international organisation, the Cochrane Collaboration.

The explicit methodologies used to determine "best evidence" were then largely established by the McMaster University research group led by David Sackett and Gordon Guyatt. According to the Wikipedia article on evidence based medicine, the term "evidence-based medicine" itself first appeared in the medical literature in 1992 in a paper by Guyatt etal. (Guyatt G, Cairns J, Churchill D, et al. [‘Evidence-Based Medicine Working Group’] "Evidence-based medicine. A new approach to teaching the practice of medicine." JAMA1992;268:2420-5. PMID 1404801)

From this point, the spread of the concept and its subsequent inclusion in professional training was rapid.

As with any other approach, evidence based medicine has its own methodological problems. However, it also has important lessons for other fields of professional practice.

PROBLEMS WITH EVIDENCE BASED MEDICINE

In my last post on evidence based medicine, I suggested that professional training focused on the transfer of existing knowledge and skills to the new professional, knowledge and skills that the professional then applied. The professional subsequently built on this base through practice, at the same time using various professional development activities to try to keep in touch with new developments.

I then posed the question what happens if that existing knowledge base is in fact wrong, looking briefly at the reasons why this proved to be the case in medicine, a discovery that had led to the development of evidence based medicine. I concluded that, as with any other approach, evidence based medicine had its own methodological problems. However, it also had important lessons for other fields of professional practice.

The Quality Movement and Quantification

In a series of posts on my personal blog, I explored some of the changes that had taken place in public administration since the war, looking at the influences on those changes.

In one of those posts I looked in part at the way in which standards, the Quality Movement and the importance of measurement had become major global influences. I also suggested that the outcomes here had not always been positive.

Evidence based medicine forms part of the global standards and quantification revolution and suffers from some of the same weaknesses. These weaknesses need to be recognised.

Problems with Evidence Based Medicine: Perception Bias

The first problem can be called simply perception bias.

Evidence based medicine is neither value nor perception free. The questions selected for test and evaluation, a process that can be very expensive, are influenced by prevailing views. Valuable alternatives may be excluded simply because they fall outside conventional wisdom. As evidence based medicine becomes the dominant mode, the effect may, as it has been in other areas, to actually narrow fields of investigation and action.

This links to a second problem, one that I have discussed before, the tendency for all professions to see answers within a frame or world view set by their profession.

A lawyer will give you a legal answer to a problem, a doctor a medical answer. If you have a back problem and see a surgeon, he/she is likely to think about surgical solutions. Go to a chiropractor with the same problem and he/she is likely to recommend spinal manipulation. So professional background helps determine the way the problem is defined, the solution applied.

This flows through into the application of evidence based approaches because the things tested are generally set within the frame of the tester. So evidence based medicine focuses on the efficacy of medical treatment and may leave non-medical alternatives aside.

Problems with Evidence Based Medicine: Causation

As part of my history honours year in my first degree I studied philosophy of history under Ted Tapp. Ted was a reflective man who required us to think about, to debate, the philosophical underpinnings of science and scientific method.

One core problem was the difference between correlation (a and b) as compared to causation (if a then b).

This problem applies in evidence based approaches. Just because a study shows an apparently strong relationship between a treatment and positive patient outcomes does not necessarily say anything about the causal relationship between the two. This has to be deduced and further tested.

Problems with Evidence Based Medicine: Problems of Epidemiological Studies

By its nature, evidence based medicine deals with large groups, populations.

As trials become larger and more complex, it becomes more difficult in statistical terms to establish significant relationships, to separate the effects of different variables. This creates another problem, the establishment of a clear relationship between the outcomes of trials at population level and subsequent application at individual level. As the Wikipedia article notes:

Critics of EBM say lack of evidence and lack of benefit are not the same, and that the more data are pooled and aggregated, the more difficult it is to compare the patients in the studies with the patient in front of the doctor — that is, EBM applies to populations, not necessarily to individuals.

This can create very real difficulties for individual clinicians, leading Tonelli to argue in The limits of evidence-based medicine that:the knowledge gained from clinical research does not directly answer the primary clinical question of what is best for the patient at hand.

Tonelli concludes that proponents of evidence-based medicine discount the value of clinical experience.

Problems with Evidence Based Medicine: Impact of the Observer

Another problem with evidence based medicine, one often seen in all evidence based approaches, is the way the observer affects the observed. This happens at several levels.

The first problem is that the simple act of participation in the trial may have some and not clearly seen impact on individual outcomes. In medicine, this is usually managed by use of a control group using a placebo. The efficacy of the treatment is then measured by the difference in outcomes between the control group and those receiving the treatment.

A second linked problem is the impact on patient behaviour of the trial itself. By their nature, clinical trials are closely managed. This means that patient compliance with the treatment routine is likely to be high.

This need not hold in subsequent clinical use since ordinary patients are more likely to fail to follow treatment processes by, for example, failing to take medication exactly as prescribed. This means that actual patient outcomes may not be as good as the trial results.

Problems with Evidence Based Medicine: Limitations in Application

A further problem is that the most rigorous gold standard approaches dictated by evidence based medicine can only be applied in narrowly defined circumstances, leaving a range of medical approaches that have to be tested by less rigorous means.

This should not matter so long as the limitations are recognised. In practice, it risks introducing two distinct distortions into the medicine and the health system. The first is the risk that investigation may be biased towards those things that can be measured through more rigorous techniques, reducing thought and investigation in areas less amenable to measurement. The second related risk is that treatment itself may become biased.

At clinician level, this links back to my earlier point about perception bias. Doctors trained in evidence based medicine may, consciously or unconsciously, come to focus in treatment terms on those things that can be measured, ruling out other less easily measured options.

This tendency may be reinforced by actions from those managing or funding the provision of health care services who may refuse to allow/pay for certain types of services notwithstanding the views of individual clinicians.

Conclusion

These problems do not detract from the potential value of evidence based medicine, but they do illustrate the need for care in application or the cure may be worse than the disease.