Longitudinal clinical data improve survival prediction after hematopoietic cell transplantation using machine learning

Allogeneic hematopoietic cell transplantation (allo-HCT) can potentially cure some individuals with hematological disorders. However, treatment-related morbidity and mortality remain major causes of therapeutic failure after allo-HCT. Organ dysfunction, opportunistic infections, and graft-versus-host disease (GVHD) are the most common causes of death in the first 100 days after allo-HCT.¹ However, organ failure can be mitigated with prompt recognition and early intervention, leading to improved long-term outcomes.^2,3 Serial evaluation of patients after allo-HCT for known modifiable risk factors or predictive biomarkers might help identify patients at high risk of organ dysfunction or other potentially lethal complications, enabling early interventions.

Several risk scores for predicting post–allo-HCT mortality, such as the HCT–comorbidity index (HCT-CI), disease risk index (DRI), disease-risk stratification system, and European Society for Blood and Marrow Transplantation risk score, have been validated for aiding clinical decision-making.^4-6 However, the predictive accuracy of these scores remains suboptimal, especially for pediatric recipients of allo-HCT.^7-10 None of them include dynamic longitudinal post–allo-HCT assessments, which could enhance the discriminatory power of the biological variables available before allo-HCT. However, incorporating numerous additional pre–allo-HCT and post–allo-HCT variables to predict the mortality risk over time would require a complex algorithm, making the statistical analysis challenging.

We sought to overcome these limitations by developing an algorithm to predict overall survival (OS) after allo-HCT that was not only based on baseline biological variables collected before allo-HCT but also incorporated the changes in patients’ clinical status after allo-HCT, thus providing a longitudinal frame of reference for risk prediction. We hypothesized that by using longitudinal data in the form of clinical and laboratory values, in addition to baseline variables, we could improve short-term and long-term mortality risk prediction for allo-HCT recipients, compared with risk prediction using baseline variables alone. We further planned to use machine learning (ML) methodology to accomplish this, using a data-driven strategy that identified underlying patterns in the observations and integrated them appropriately and precisely, instead of using a priori assumptions or predefined statistical methods.

This was a retrospective study conducted to model OS after allo-HCT for pediatric patients. The primary objective was to predict OS at 100 days after allo-HCT. Secondary objectives included predicting OS at 1 and 2 years after allo-HCT. For these respective objectives, mortality was defined as death from any cause by 100 days, 1 year, and 2 years after allo-HCT. Each end point of interest was treated as a binary outcome. To leverage patients’ longitudinal clinical information before and after allo-HCT for OS prediction, the observation window for incorporating repeated clinical measurements in the model was set from 30 days before allo-HCT to 30 days afterwards. Clinical measurements outside this window were not included in this model. The predictive timeline is illustrated in Figure 1. The study was approved by the St Jude Children’s Research Hospital (St Jude) Institutional Review Board and followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis reporting guideline for prediction algorithm development.¹¹ It was conducted in accordance with the Declaration of Helsinki.

Patients were eligible for inclusion in the internal training and validation cohorts if they underwent their first allo-HCT at St Jude between 2000 and 2020. An external validation cohort consisted of patients undergoing their first allo-HCT at Memorial Sloan Kettering Cancer Center (MSKCC) between 2012 and 2020. Additional inclusion/exclusion criteria and details of the numbers of patients who met the various criteria are shown in Figure 2. Because this was a pragmatic study, the sample size was determined by the number of eligible patients who underwent allo-HCT during the study period and not by a priori power calculations. Of the 922 patients who underwent allo-HCT at St Jude between 2000 and 2020, 738 met the eligibility criteria and were included in the study cohort.

Baseline variables, including recipient and donor characteristics, as well as disease-related and allo-HCT–related factors, were extracted from electronic medical records (EMRs) and a curated institutional transplant database. Baseline variables are summarized in Table 1. Multiple imputation using chained equations¹² was performed on the baseline variables to address potential bias and increased variance in parameter estimates resulting from unknown predictors. All longitudinal variables were extracted from the EMRs, except for GVHD-related data, which were extracted from the curated transplant database. Nonnumeric longitudinal variables, such as microbial surveillance, imaging reports, and measurement device descriptions, were excluded from the analysis, given their subjective nature. To model the trajectory of the longitudinal data over the observation window, the repeated measurements were summarized in 6 summary statistics.¹³ Longitudinal summary statistics with missingness after discretization were removed. The 46 longitudinal variables that were initially considered, and their summary statistics that were finally included in the analysis are listed in supplemental Table 1.

Naïve-Bayes, which is a supervised classification technique used to classify subjects by assigning class labels based on conditional probability, was used to model OS.¹⁴ Details of the model development and validation are provided in the supplemental Methods. The predictive performance of the established naïve-Bayes model was first validated on the internal validation data set based on several performance metrics described in the supplemental Methods. External validation of our established predictive models was then performed using the validation data set from MSKCC.

Two separate sensitivity analyses were performed to test the robustness of our prediction model pipeline. First, because patients with active disease or primary refractory leukemia who are included in the current St Jude data set are not considered candidates for allo-HCT at many institutions, the predictive model was retrained by excluding these patients and those with missing disease status from the St. Jude data set. Second, to incorporate organ function details available at baseline or before allo-HCT, another model was created that incorporated any laboratory data collected between days −30 and −7 as a part of the baseline variables. This new model emulates HCT-CI, which incorporates organ function assessments available at the time of allo-HCT. The performance metrics were then recalculated for the St Jude validation data set, using the models established in these 2 sensitivity analyses.

The baseline characteristics and survival outcomes of patients in the entire St Jude cohort (N = 738), the St Jude training data set (n = 517) and St Jude validation data set (n = 221), and the MSKCC cohort (N = 218) are listed in Table 1. The median follow-up duration for the entire cohort was 4 years (range, 0.1-21.7 years). The St Jude training and validation data sets were similar with respect to the baseline variables. There were several significant differences between the St Jude and MSKCC cohorts: the St Jude cohort had greater proportions of White recipients; recipients with mismatched related donors; and patients who received a transplant for acute leukemia or had active disease at the time of allo-HCT, received reduced-intensity conditioning, or received a bone marrow graft (P < .05 for all comparisons). At each observation time point, the proportion of deceased patients was greater in the St. Jude cohort than in the MSKCC cohort.

We created 3 naïve-Bayes–based models using the St Jude training data set for predicting the 100-day, 1-year, and 2-year OS: (1) a base-only model with all the baseline variables; (2) a longitudinal-only model with only the longitudinal variables among the top 50 selected predictors; and (3) the full model with the top 50 selected predictors, including both baseline and longitudinal variables. The performance of these ML models was first internally validated on the St Jude validation data set and then externally validated on the MSKCC data set. Figure 3 illustrates the mean area under the curve (AUC) values, obtained by bootstrapping 500 times on the St Jude validation data set, based on the predictions of the 3 models for the different end points. The longitudinal-only and full models significantly outperformed the base-only model in predicting all 3 end points of interest, with substantially higher AUC values that were all significantly different (Figure 3).

To evaluate further the performance of the models in predicting OS at different time points, we plotted the survival probabilities of patients classified as deceased or alive by the models based on the internal validation data set, using Kaplan-Meier plots (Figure 4). When predicting OS at 100 days, the longitudinal-only and full models improved the stratification of the patients classified as deceased or alive, with the P values obtained from the log-rank tests being 6.16 × 10⁻¹¹ and 5.64 × 10⁻¹³, respectively, significantly lower than that of the base-only model (P = 8.32 × 10⁻²; a smaller P value indicates better stratification). The full and longitudinal models similarly outperformed the base-only model by achieving better stratification of patients classified as deceased or alive at 1 and 2 years.

Figure 5 shows a detailed ranking based on the Shapley Additive exPlanations (SHAP)¹⁵ values of the predictive importance of the baseline and longitudinal predictors in the full model when predicting different end points of interest. Only 1 baseline variable (transplantation year) and a few pre–allo-HCT laboratory values, such as mean serum albumin, total protein, bilirubin, calcium levels, slope of the change in blood urea nitrogen, and total protein and albumin, were among the important variables for predicting 100-day OS. All other predictors that ranked higher in terms of SHAP values for the 100-day OS prediction were longitudinal variables collected after allo-HCT. Although several other baseline variables, such as graft source and donor, degree of HLA match, and disease diagnosis and status, emerged as being among the more important variables for predicting longer-term OS (at 1 and 2 years after transplant), the longitudinal laboratory values remained among the most important predictors of survival.

Detailed results of the sensitivity analyses are provided in the supplemental Results. Briefly, in the first sensitivity analysis, after excluding patients with active or refractory disease or missing disease status from the model training, the full model performed the best of the 3 in predicting 100-day OS, and it outperformed the base-only model in predicting 1-year and 2-year OS (supplemental Figures 1 and 2). In the second sensitivity analysis, the predictive ability of the new base-plus model, which incorporated the latest available laboratory data collected between days −30 and −7 as a part of the baseline variables, surpassed that of the base-only model. However, the longitudinal-only and full models, which include more comprehensive longitudinal information, still outperformed the base-plus model in predicting all 3 end points of interests (100-day, 1-year, and 2-year OS) (compare supplemental Figures 3 and 4 with Figures 3 and 4).

Detailed results of the external validation are provided in the supplemental Material. Briefly, the full model outperformed the others in predicting 100-day and 1-year OS in the MSKCC cohort (supplemental Figures 5 and 6). Although the full-model AUC was slightly smaller than that obtained with the base-only model when predicting long-term OS (ie, 2-year OS), the overall results are consistent with our findings based on the St Jude validation cohort.

A demonstration of the validated model is available at https://sjbiostat.shinyapps.io/pedsHCT/.

This cohort study showed that ML models developed by incorporating longitudinal data collected from 30 days before to 30 days after allo-HCT can accurately predict short-term (100-day), intermediate-term (1-year), and long-term (2-year) OS of pediatric patients undergoing their first allo-HCT. When compared with models constructed from baseline variables alone, the ML models incorporating longitudinal data were better at discriminating between patients predicted to be deceased or alive at the given time points. Importantly, unlike traditional prognostic tools that use fixed variables for risk assessment, our model incorporates dynamic variability in the clinical and laboratory data, which is critical for improving mortality prediction.

Risk stratification of patients based on biological and clinical factors to predict treatment outcomes and guide treatment decisions is a cornerstone of modern oncology. Many such risk-prediction models are currently used in allo-HCT practice. The HCT-CI assigns weights to a composite of 17 baseline organ function criteria present before allo-HCT to predict nonrelapse mortality after allo-HCT.⁵ Adding more comorbidities while adapting the HCT-CI for pediatric patients did not enhance its discriminative capacity.¹⁶ Although adding pre–allo-HCT laboratory assessments such as ferritin, albumin, and platelet count augmented the HCT-CI,¹⁷ it only improved the concordance statistic of the score marginally.¹⁸ These observations not only suggest the limitations of using only the fixed baseline variables at the time of allo-HCT for risk prediction but also indicate that including biomarkers might be key to improving prediction capabilities. The DRI and disease-risk stratification system use a patient clinical status agnostic approach to risk-categorize patients based on the disease diagnosis, clinical status, or molecular and cytogenetic data.^6,19 The European Society for Blood and Marrow Transplantation risk score combines some components of the HCT-CI (age) and DRI (disease stage) and adds some new allo-HCT–related factors (donor details) to the model.⁴ Although the score remains predictive of outcomes, the incidence of treatment-related mortality is strongly influenced by the allo-HCT conditioning regimen intensity.⁴ Notably, none of these risk scores incorporate the impact of the conditioning regimen intensity on the patients into the prediction algorithm, even though the conditioning regimen is well-known to be an important predictor of post–allo-HCT outcomes.²⁰ Additionally, their predictive accuracy is limited because of the inherent shortcomings of the components included and the statistical methodology used.^7-10 

The advent of artificial intelligence and ML offers the opportunity to integrate health data from various sources into personalized patient risk assessments. Many ML methods are readily applicable to complex clinical scenarios, including high-dimensional or even unstructured data involving complex interactions. Early attempts to apply ML algorithms to EMR data have demonstrated their power to accurately predict short-term mortality in general medicine^21,22 and oncology settings.^23-26 Similar attempts have been made to use large data sets to predict short-term mortality and complications such as GVHD after allo-HCT in adult patients.^13,27-30 But to our knowledge, none have integrated longitudinal assessments in outcome prediction. Here, we created a naïve-Bayes model that incorporated numeric longitudinal laboratory values and clinical data to supplement the predictive ability of baseline demographic, disease-related, and allo-HCT–related factors that are traditionally used for survival prediction. Naïve-Bayes was chosen for the model construction mainly because it does not require extensive training data and because it is highly scalable with numerous predictors.¹⁴ It is fast and can be used to make real-time predictions, which will be crucial for its future implementation as a clinical decision-making tool.

Adding the longitudinal variables significantly improved the predictive ability of the model compared with using baseline variables alone. The AUC values of the receiver operating characteristic curves were consistently higher for the models incorporating the longitudinal variables than for the model including baseline variables only. The improvement in OS prediction with the inclusion of longitudinal information does, however, tend to decrease over time from the 100-day prediction to the 2-year prediction. This decrease in the discriminatory power of the models incorporating the longitudinal variables at later time points after allo-HCT is probably due to the relatively short window of observation for the longitudinal variables, which was limited to 30 days after allo-HCT for all prediction time points in this study. By extending the window of observation of longitudinal variables to, perhaps, 6 months after allo-HCT or later, we may further improve the prediction of OS at 1 and 2 years after allo-HCT.

By examining the SHAP values of individual variables, we identified significant predictors that contributed to a given patient being classified as deceased or alive. Many variables with known prognostic implications, such as disease diagnosis, disease status, donor-recipient HLA matching, graft source, and serum albumin and ferritin levels before allo-HCT, were highly predictive of outcomes in the different models. Unsurprisingly, variables such as markers of liver function (serum albumin, total protein, and serum bilirubin) and kidney function (blood urea nitrogen), serum electrolytes, lactate dehydrogenase levels, and blood glucose, collected before or after allo-HCT, also proved to be highly predictive. When the SHAP values were compared across prediction time points, the SHAP values of the different longitudinal measurements, most of which were collected after allo-HCT, were much higher than those of the baseline variables, suggesting that the longitudinal variables are much more important for predicting OS. In certain instances, risk factors that were not explicitly indicated in the model also were noted to be highly predictive. For example, the number of tests performed to check for direct bilirubin after allo-HCT ranked very highly in the models predicting 100-day and 1-year mortality. Veno-occlusive disease or sinusoidal obstruction syndrome of the liver is a known complication of allo-HCT that is associated with high mortality. One of the signs of development of veno-occlusive disease in a patient after allo-HCT is rising serum bilirubin. It is likely that clinicians caring for patients at risk of developing veno-occlusive disease assessed their direct bilirubin more often than patients who were not at such risk. Even though veno-occlusive disease was not explicitly included as a risk factor for mortality in this model, the ML algorithm was able to identify this trend and thus included the number of tests for direct bilirubin after allo-HCT as a prominent predictor in the model. Although some of these covariates (such as the number of tests performed for direct bilirubin assessment or the standard deviation of CO₂ and chloride which may indicate acid-base imbalance) might be quite obvious clinically, some others (such as number of tests for magnesium after allo-HCT) may not be as biologically relevant immediately. This is one of the main strengths as well a noted limitation of data-driven methods: the algorithms can identify predictors that otherwise would not be selected based on our current knowledge, but sometimes the underlying biological mechanism may not be apparent. More importantly, these observations suggest not only that dynamic longitudinal variables can supplement the predictive ability of the static baseline variables but also that these long-term measurements reflecting changes in clinical status have much more discriminatory power to identify patients who are likely to have poor outcomes, even at distal time points. This provides an excellent window of opportunity during which intensive supportive-care measures to prevent further organ dysfunction or improve organ function might be instituted, thereby improving patient outcomes. Our future endeavors will focus on predicting these adverse events, such as organ dysfunction, and complications such as GVHD after allo-HCT, which eventually lead to mortality, therefore helping clinicians institute appropriate measures to mitigate these adversities.

The results of this study must be considered in light of the following limitations. First, this was a retrospective analysis based on data routinely collected during patient care. Accordingly, several baseline variables were missing for some patients, and these had to be imputed to generate a complete data set for training and validation. Although using multiple imputations, rather than a single “best guess,” might have enhanced the model calibration and avoided over-fitting on imputed values, any nonrandom missingness may still bias results. Second, the model was developed on a data set from a single academic quaternary-care referral center, which may limit its generalizability. However, we validated our results on an external data set with very different patient characteristics (from MSKCC) with similar prediction accuracy, highlighting the model’s robustness. Even though the St Jude and MSKCC data sets differed significantly in several critical baseline variables, the model was highly discriminatory when applied to the MSKCC data set, underscoring the fundamental principles on which the model is based. Third, this model did not include recipients of allogeneic umbilical cord blood grafts or autologous hematopoietic stem cell rescue or those who received an allo-HCT for a solid tumor as the indication. There were too few patients in the training cohort who met these criteria and, hence, limited outcome information can be robustly and reliably extracted from these covariates. Hence, these patients were excluded from the model development, and the results may not be generalizable to them.

To our knowledge, this is the first report of ML being used to identify allo-HCT recipients at risk of short-term or long-term mortality by incorporating longitudinal data collected during clinical care before and after allo-HCT. This proof-of-concept study highlights the importance of incorporating longitudinal post–allo-HCT data in these prediction algorithms. Prospective validation of this model will enable the development of a clinical decision support tool to complement clinical intuition and enable clinicians to deliver evidence-informed care based on real-time patient data and thereby improve clinical outcomes.

The authors thank Keith A. Laycock for scientific editing of the manuscript.

Y.Z., J.S., D.C.S., C.L., Y.S., S.S.M., S.N., B.M.T., L.T., and A.S. are supported for their work at St Jude Children’s Research Hospital by the American Lebanese Syrian Associated Charities and National Institutes of Health/National Cancer Institute grant (P30 CA021765). R.S. was supported by a Memorial Sloan Kettering Cancer Center Core grant (P30 CA008748) from the National Institutes of Health/National Cancer Institute.

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Contribution: Y.Z., J.S., L.T., and A.S. conceptualized the study, developed methods, implemented computational algorithms, performed data analysis, interpreted results, and wrote the manuscript; D.K. cleaned and processed the data and helped with the implementation of the study procedures; all other authors provided inputs to the study design and critically reviewed the data, the results, and the manuscript draft; and all authors approved the final version of the manuscript.

Conflict-of-interest disclosure: A.S. has received consultant fees from Spotlight Therapeutics, Medexus Inc, Vertex Pharmaceuticals, Sangamo Therapeutics, and Editas Medicine; is a medical monitor for a Resource for Clinical Investigations in Blood and Marrow Transplantation (RCI BMT) clinical trial for which he receives financial compensation; received research funding from CRISPR Therapeutics and honoraria from Vindico Medical Education; and is also the St Jude Children’s Research Hospital site principal investigator for clinical trials of genome editing for sickle cell disease sponsored by Vertex Pharmaceuticals/CRISPR Therapeutics (NCT03745287), Novartis Pharmaceuticals (NCT04443907), and Beam Therapeutics (NCT05456880) (the industry sponsors provide funding for the clinical trials, which includes salary support paid to the investigator’s institution and A.S. has no direct financial interest in these therapies). The remaining authors declare no competing financial interests.

Correspondence: Akshay Sharma, Department of Bone Marrow Transplantation and Cellular Therapy, St Jude Children’s Research Hospital, 262 Danny Thomas Pl, MS 1130, Memphis, TN 38105; email: akshay.sharma@stjude.org.