The Evidence-Based Analysis of Treatment for Chronic Myeloid Leukemia: An Introduction to Its Methods and Clinical Implications

IN THIS ISSUE OF BLOOD, a panel sponsored by the American Society of Hematology (ASH) publishes its analysis of data on selected modalities used for treatment of the chronic phase of chronic myeloid leukemia (CML).1 The text is long, and the data presented are complex and arrayed in elaborate tables. The strengths and weaknesses of the scientific evidence are discussed in detail, and the recommendations are carefully worded. All of these features make it a complex document for the reader. However, each of these features is critical to the goal of this project: to achieve an objective assessment of the effectiveness and risks of the available treatments.

Systematic reviews featuring this comprehensive approach to evaluate the literature have become a cornerstone of “evidence-based medicine.”2,3 The benefit of systematic, evidence-based reviews is that they inform practitioners about the quality of evidence supporting what we do in medicine. In the case of CML, as is often the case, widely held beliefs about the effectiveness, superiority, and safety of treatments are challenged when exposed to critical appraisal of the evidence. Clinicians can continue to use these treatments, but they do so with a better understanding of the scientific basis. Systematic reviews aid in practice guideline development by critically appraising and summarizing the results of primary research.4 Identifying gaps in the evidence is also useful to researchers, and to funding agencies, to help set priorities for future research.

Evidence-based practice guidelines have some limitations. Good evidence is lacking for much of what is done in medicine, yet clinicians need practical advice on how to manage patients even when high-quality evidence is lacking. Evidence-based reviews are relatively expensive, take months to years to complete, and may become outdated when new evidence becomes available. Perhaps of greatest concern to practitioners, the conclusions of evidence-based reviews and guidelines can be harmful if they are cited by health plans or government agencies as grounds for withholding coverage and referrals, or if they are used unfairly for judging the quality of care or for malpractice litigation. Some physicians believe that these risks argue against participating in guideline development, while others contend it is more dangerous to allow forces outside the specialty to write and impose guidelines.

These issues took center stage in the work of the CML panel. The report it produced1 is the product of 3 years of intensive effort by an international panel of experts, which conducted a systematic search of the literature, using standardized methods to judge the quality of over 200 studies. The data from this analysis are contained in seven tables. Table 1 is a statement of predefined rules that guided the recommendations. Tables 2 through 7 contain a summary of the published data. Readers should appreciate both the scientific limitations of the literature, which includes uncontrolled observational studies, as well as the absence of key data in many of the cited publications. For example, many studies do not report fundamental data on their patient population, such as age, Philadelphia chromosome status, and duration of follow-up. Some data on long-term survival are based on observations on very few patients. These features help to explain the extreme variation between studies in the reported survival benefits and the incidence of toxicities and complications of interferon treatment and allogeneic bone marrow transplantation (BMT).

The methodologic limitations in the evidence had different meaning for panel members, some of whom were the lead authors of the studies under review. Proponents of particular treatments, who believed strongly in their value, were less concerned about these limitations or the absence of evidence from randomized controlled trials than were panel members who served as “methodologists.” When evidence was lacking, some panel members felt a duty to offer clinicians firm recommendations anyway, even if based on opinion, whereas others believed that it was preferable to avoid such recommendations and to state only that there was insufficient evidence. Some panel members believed that the evidence should drive the recommendations, while others preferred to adjust the guidelines to reflect current practices and opinion. Much of the 3 years involved in the production of the CML guideline was spent arbitrating these disagreements.

Some of these difficulties reflected the composition of the panel, which consisted largely of CML investigators. Some guideline panels of other specialty societies and government agencies intentionally limit the participation of renowned experts and of authors of the studies under review to avoid these difficulties and to allow reviewers to approach the evidence free of preconceived viewpoints. Guideline panels of the American College of Physicians, US Preventive Services Task Force, and other groups, which review evidence for topics that cut across multiple specialties, are composed of individuals who are skilled in the critical appraisal of evidence but who may not even belong to a relevant specialty. Critics of this approach worry that unfamiliarity with the topic and lack of experience with direct care of patients with the condition limits the capacity to produce clinically relevant guidelines. Sensitive to this concern, Dr Richard Silver, the Chair of the CML panel, chose leading clinical experts to serve as members, but he also appointed a statistician and a practice guideline methodologist. This set the stage for an interesting dilemma. Many of the studies reviewed by the CML panel were considered excellent by the clinical experts but of poor quality by the clinical research methodologists. Whose views of the evidence should prevail? Experts on CML, many of whom had conducted the studies, surely knew the data best. On the other hand, having conducted the studies, could they objectively review their own work? Epidemiologists and biostatisticians may be more dispassionate and are probably more qualified to evaluate the validity of Kaplan-Meier survival curves and statistical power calculations. But, without background in CML, can they fully interpret all the information in the reports? Without having treated CML patients, should they influence the recommendations?

What is the practicing hematologist to make of all of this? What is the value of the CML review (and of other evidence-based reports) in the real world of patient care, where decisions cannot be postponed because of limited data? Clinical decisions are based on multiple factors, only one of which is the evidence from clinical research. Practitioners of evidence-based medicine emphasize three key elements of clinical decision-making: clinical experience, research evidence, and patient preferences.5 Thus, beyond research evidence, choices about BMT, interferon, or other options for patients with CML are influenced by clinical experience, including assessment of the patient’s history and risk factors, as well as by patient expectations and preferences, and by restrictions imposed by health care systems or third-party payers. To the extent that clinical decisions incorporate scientific evidence, systematic reviews serve a critical purpose by casting a spotlight on the strength and weaknesses of the evidence. Clinicians must still consider other factors in clinical decision-making, but with more information on the quality of the research evidence, they are better able to correctly inform their patients about the benefits and risks of the available treatment options.

The fact that current data do not allow more definitive recommendations is perhaps the most important finding of the CML analysis. This is an important message for both clinicians and researchers about the limitations of existing clinical research evidence on CML treatment. With a few notable exceptions, the CML panel found that the studies lacked basic documentation. Inclusion criteria were vague and applied inconsistently. Sample sizes were generally small, providing inadequate statistical power to evaluate differences in outcomes, and wide 95% confidence intervals for the estimates of treatment effect. Treatment protocols were not adhered to systematically, with large drop-out rates. These deficiencies pose a serious problem as the health care system applies increasing scrutiny to the quality of evidence. Interventions that are not supported by well-designed studies are less likely to be paid for. The findings of the CML report are a clarion call to researchers to use rigorous study designs and to observe greater scientific diligence in data collection and adherence to protocol.

The CML review brings into sharp focus the fact that all treatment options for CML involve difficult tradeoffs between benefits and harms, and that the choice of which option is “best” is influenced by subjective value judgments. Reflecting a growing trend in practice guidelines,6 the CML report argues that the patient should have the opportunity to decide for himself or herself, based on personal preferences, which option is best. It is a recommendation that some CML panel members, accustomed as experts to telling patients what is best, found discomforting. Yet the era of “shared decision-making” is upon us,6,7 as empowered patients informed by the Internet and other resources seek more knowledge about their options, and play an active role in decisions about their care.

The section of this document outlining recommendations required the most effort and greatest diplomacy. For the reader, it represents the best interpretation of the current research evidence. Evidence from randomized controlled trials demonstrates that treatment with interferon is superior to treatment with hydroxyurea, but there are more side effects with interferon. There are no randomized controlled clinical trials to document the comparative risk and benefit of BMT. Observational studies suggest that BMT does increase the probability of long-term survival, but also document a significant short-term risk of complications or death from the procedure. These observational studies involve significant patient selection. Therefore, at present, definitive conclusions cannot be made about the relative benefit and risk of BMT versus interferon in patients with CML. The trade-offs between higher early mortality with BMT but a potential long-term survival advantage, as well as the side-effects of interferon therapy, underscore the importance of incorporating patient preferences into the clinical decision about treatment options. The evidence-based analysis by the CML panel makes a major contribution by bringing these issues into focus. There is an urgent need for a definitive randomized clinical trial, incorporating sufficient long-term follow-up, to evaluate the relative benefits and risks of BMT versus the most promising alternative treatment in patients with CML. The report of the CML panel creates sufficient “clinical equipoise”8to undertake such a trial.