TO THE EDITOR:

The American Society of Hematology (ASH) annual meeting is often considered the premier international conference in hematology. When “positive” results from single-arm phase 2 trials are presented at ASH, one would expect that these herald therapeutic advances, or at a minimum, follow into a larger, potentially confirmatory, randomized (phase 3) trial. However, a survey of ASH abstracts reporting positive results in acute myeloid leukemia (AML) indicated that these positive abstracts, accounting for the majority of abstracts, had limited predictive value for subsequent clinical utility.1 As phase 3 trials are initiated to further investigate clinical benefit, proceeding (or not) from a positive abstract to a phase 3 study is a topic that warrants investigation. Across all hematological malignancies, it is currently unclear how often positive (or negative) abstracts lead to phase 3 trials, as are the reasons why or why not. Here, we hypothesized that most ASH abstracts reporting on phase 2 trials in oncological hematology are positive, and that the majority of drugs in these trials will not reach investigation in phase 3. Finally, because abstracts are necessarily preliminary, subsequent peer-reviewed publication is essential for correct dissemination of trial results. Therefore, we additionally assessed to what extent positive, rather than negative, studies are subsequently published as full-length peer-reviewed papers, potentially leading to “publication bias.”

We downloaded all 2013 through 2015 ASH abstracts (N = 17 251), using a web-scraper-based approach. This timeframe allowed at least 5 years between the abstract presentation and publication as full-length paper or advancement to phase 3. Abstracts (n = 1100) were selected based on presence of “phase 2” and all possible spellings thereof, in the title and/or abstract text. We excluded benign hematology, biomarker trials, phase 1/2 trials that reported only phase 1 data, and trials without efficacy results. If multiple abstracts reported or updated on the same study, only the most recent abstract was included. These processes yielded 368 abstracts, covering lymphomas (35%), multiple myeloma (21%), AML (12%), chronic lymphocytic leukemia (11%), myelodysplastic syndrome (8%), acute lymphoblastic leukemia (5%), chronic myeloid leukemia (3%), and other (5%). Abstracts were scored “positive,” “negative,” or “inconclusive.” Criteria for a positive abstract were words/phrases such as “encouraging,” “promising,” “could represent a novel therapeutic option,” and “warrants investigation in a randomized trial.” “Negative” abstracts included terms such as “does not support further research” and “demonstrates no clinical activity.” The remainder were scored as inconclusive. Uncertainty was resolved by discussion between the authors.

We scored 292/368 (79.3% 95% confidence interval [CI], 75-84) abstracts as positive, 37/368 (10.1% 95% CI, 7-14) as negative, and 39/368 (10.6%; 95% CI, 8-14) as inconclusive (Table 1).

We then searched papers indexed in PubMed using names of the primary or senior authors and 1 to 4 title keywords, with the most recent update on 1 August 2021. We found that 302/368 abstracts (82.1%; 95% CI, 78-86) were published as full-length papers in peer-reviewed journals (Table 1; Figure 1A). The abstract conclusion was not significantly associated with a full-length publication. Abstracts published as full-length papers had larger sample sizes than nonpublished abstracts (P = .001). Studies were most often published in British Journal of Haematology (39/302 [12.9%]) and Blood (34/302 [11.3%]). A total of 91.2% (31/34) of the studies published in Blood were positive. British Journal of Haematology published significantly more negative studies than Blood (Figure 1A; 26% vs 5.9%, Fisher exact P = .02). Positive studies were published more often in journals with higher impact factors (mean 2020 impact factor 18.2 for positive vs 9.2 for negative studies, P < .001; Figure 1B). Differences between the abstract conclusion and later peer-reviewed conclusion were infrequent: 6.2% of positive ASH abstracts changed to a negative conclusion in the peer-reviewed publication and vice versa.

Figure 1.

Details of published phase 2 ASH abstracts in peer-reviewed journals and reasons for discontinuation to phase 3 trials. (A) Overview of journals publishing phase 2 studies as presented at ASH from 2013 through 2015 according to the conclusion as presented in the publication. Journals with at least 2 published ASH abstracts are displayed. (B) Violin plot of 2-year 2020 impact factors of journals publishing positive, negative, and inconclusive studies initially presented at ASH according to the conclusion in the publication. (C) Overview of reasons for clinical research not progressing to randomized phase 3 despite of positive phase 2 results.

Figure 1.

Details of published phase 2 ASH abstracts in peer-reviewed journals and reasons for discontinuation to phase 3 trials. (A) Overview of journals publishing phase 2 studies as presented at ASH from 2013 through 2015 according to the conclusion as presented in the publication. Journals with at least 2 published ASH abstracts are displayed. (B) Violin plot of 2-year 2020 impact factors of journals publishing positive, negative, and inconclusive studies initially presented at ASH according to the conclusion in the publication. (C) Overview of reasons for clinical research not progressing to randomized phase 3 despite of positive phase 2 results.

Close modal

Fifty-three percent of regimens described in positive abstracts did not proceed to phase 3 studies as reported in clinicaltrials.gov, and the corresponding figure for positive full-length papers was 47%. Progression to phase 3 was most common in chronic myeloid leukemia (86%) and lowest in lymphomas (46%). Twelve regimens described as negative and 9 as inconclusive in full-length papers proceeded to phase 3. Regimens proceeding to phase 3 trials were based on phase 2 trials with larger sample sizes as reported in the full-length paper (median 58; interquartile range, 40-87 vs 42, interquartile range, 28-69 for regimens that did not progress; Wilcoxon rank sum P < .001). To explore reasons for discontinuation, we sent questionnaires (supplemental Data) to the authors of positive studies (based on full-length papers) not prompting phase 3 trials. Three attempts were made to reach the authors; 51% (54 of 105 reachable authors) responded. The respondents were asked to confirm that there was no subsequent phase 3 trial and to select reasons why a randomized phase 3 trial was not undertaken. Failure of positive phase 2 trials to proceed to phase 3 was due to a decision by the pharmaceutical company (43%), lack of any intent to study the drug in phase III in the first place (41%), insufficient funding (35%), insufficient efficacy (despite the “positive” conclusion in the publication; 33%), and safety concerns (4%) (Figure 1C). Additional reasons were the availability of newer regimens, the rarity of the disease, or regulatory approval already obtained after phase 2.

Elaborating on the previous AML survey that inspired this research,1 we assessed clinical utility of AML therapeutics in our dataset. Thirty-five abstracts reported on 24 individual drugs, of which midostaurin and venetoclax received regulatory approval, and quizartinib was positive in the phase 3 trial.2 This suggests more favorable progression rates to phase 3 compared with previously reported data, in which only 1 of 37 drugs described in positive abstracts migrated into clinical practice.1 

Remarkably, one-third of regimens did not proceed to phase 3 at least in part because of insufficient efficacy, despite most recent presentation (as full-length paper) as promising. Reasons for discontinuation after a positive phase 2 trial is understandable in cases of funding constraints or safety concerns. However, numerical overestimation of initial results,3 or overly optimistic interpretation, not tempered through peer review, may be an additional explanation for these findings. These data therefore stress the importance of a critical reading of papers (and abstracts) with attention to issues such as selection bias and clinical relevance of reported “responses.” The discontinuation of promising novel regimens per decision of pharmaceutical companies may also be problematic: although some such decisions may have reflected (unreported) funding constraints, there also is an unfortunate implication that shareholder welfare can prevail over patients’ best interests. Similarly, patient participation in research never intended to continue to phase 3 is questionable given that outcomes in single-arm phase 2 trials are often poor predictors of outcomes in randomized phase 3 trials.4,5 Because conducting phase 2 trials seems essential for career development of junior researchers, an ethical tension between the patient’s and researcher’s best interests arises.

These data confirm earlier findings that most abstracts presented at ASH are still represented as positive. It is reassuring that the large majority of ASH abstract are subsequently published as full-length papers, and overall, the conclusions are in line with each other, suggesting that early interpretation of data are reliable. Fortunately, positive and negative studies are represented equally often in peer-reviewed journals, not indicating any “direct” publication bias. However, positive studies are published more often in high-impact journals, suggesting an "indirect" publication bias. Because truly negative studies may be more rigorous and accurate than positive studies, high-impact journals should be encouraged to publish “negative” studies to ensure equal exposure.

Although further investigation is needed into why therapeutics in positive phase 2 trials do not progress to phase 3 trials, it is encouraging that nearly one-half of the regimens presented in positive ASH abstracts are steered toward phase 3, considering the resources needed for randomized studies.2 Further improving research practice and ethics, combined with trial designs that smartly test multiple “promising” agents,2,6 as well as synthesizing evidence from additional resources such as registries or expanded access programs,7 will further advance the development and implementation of novel therapeutic options across hematological malignancies.

The authors thank all respondents to the questionnaire for providing the information necessary for this research and the reviewers for providing constructive suggestions. The authors also thank Noor Gieles for proofreading the manuscript.

We dedicate this work to Elihu H. Estey, our mentor, collaborator, and friend, who unexpectedly died during the process of writing and revising this manuscript.

Contribution: D.G.J.C. and T.B.P. conceived the research; G.J.O., J.M.R., and E.H.E. supervised the research; D.G.J.C. and T.B.P. collected and analyzed data and drafted the manuscript; D.G.J.C. prepared the figures; all authors revised the manuscript and approved the final version for initial submission; all authors participated in the revision of the manuscript; and D.G.J.C., T.B.P., G.J.O., and J.M.R. approved the final version of the manuscript.

Conflict-of-interest disclosure: The authors declare no competing financial interests.

Elihu H. Estey died on 8 October 2021.

Correspondence: David G. J. Cucchi, Department of Hematology, Amsterdam UMC, Vrije Universiteit Amsterdam, Cancer Center De Boelelaan 1117, 1081HV Amsterdam, The Netherlands; e-mail: d.cucchi@amsterdamumc.nl.

The online version of this article contains a data supplement.

1.
Walter
RB
,
Appelbaum
FR
,
Tallman
MS
,
Weiss
NS
,
Larson
RA
,
Estey
EH
.
Shortcomings in the clinical evaluation of new drugs: acute myeloid leukemia as paradigm
.
Blood.
2010
;
116
(
14
):
2420
-
2428
.
2.
Cucchi
DGJ
,
Polak
TB
,
Ossenkoppele
GJ
, et al
.
Two decades of targeted therapies in acute myeloid leukemia
.
Leukemia.
2021
;
35
(
3
):
651
-
660
.
3.
Beyar-Katz
O
,
Rowe
JM
,
Townsend
LE
,
Tallman
MS
,
Hadomi
R
,
Horowitz
NA
.
Published abstracts at international meetings often over- or underestimate the initial response rate
.
Blood.
2017
;
129
(
16
):
2326
-
2328
.
4.
Gyawali
B
,
D’Andrea
E
,
Franklin
JM
,
Kesselheim
AS
.
Response rates and durations of response for biomarker-based cancer drugs in nonrandomized versus randomized trials
.
J Natl Compr Canc Netw.
2020
;
18
(
1
):
36
-
43
.
5.
Zia
MI
,
Siu
LL
,
Pond
GR
,
Chen
EX
.
Comparison of outcomes of phase II studies and subsequent randomized control studies using identical chemotherapeutic regimens
.
J Clin Oncol.
2005
;
23
(
28
):
6982
-
6991
.
6.
Duetz
C
,
Cucchi
DGJ
,
Polak
TB
, et al
.
The wider perspective: twenty years of clinical trials in myelodysplastic syndromes
.
Br J Haematol.
2021
;
196
(
2
):
329
-
335
.
7.
Polak
TB
,
van Rosmalen
J
,
Uyl-de Groot
CA
.
Expanded access as a source of real-world data: an overview of FDA and EMA approvals
.
Br J Clin Pharmacol.
2020
;
86
(
9
):
1819
-
1826
.

Supplemental data

Sign in via your Institution