External Validation of Prognostic Models in Terminally Ill Patients,

Over one million Medicare beneficiaries receive hospice care annually. However, besides the well documented advantages of hospice, many Americans do not enjoy maximum benefit from the hospice care. The fundamental reason for this is related to the inappropriate and poorly timed referral of terminally ill patients to hospice. As a result, many patients die within a few days of referral, while some live many years after the referral was made. Improvement in the accuracy of prognosis translates into superior quality of care. Predictions based on statistical modeling have been shown to be superior to physicians' prognostication. However, very few of these statistical models have been externally validated in terminally ill patients. Here we report the external validation of 5 most commonly used prognostication models in a cohort of terminally ill patients: 1) declining exponential approximation of life expectancy (DEALE) 2) study to understand prognoses and preferences for outcomes and risks of treatments (SUPPORT), 3) adjusted palliative performance scale (PPS), 4) adjusted Karnofsky performance scale index (Karnofsky) and 5) adjusted eastern cooperative oncology group performance status (ECOG).

We retrospectively extracted data from 590 deceased patients enrolled in Tampa Bay Lifepath Hospice and Palliative Care starting January 2009 and going backwards to validate the prognostic models. Two research assistants extracted all data necessary to populate the model variables and two faculty members randomly checked 25% of the data for accuracy. The models were tested against observed survival duration. PPS, Karnofsky and ECOG risk scores were predicted using a flexible family of Royston-Parmar parametric models and adjusted for age, gender and presence of cancer. We utilized several metrics to assess the performance of these models. Specifically, we used the Brier score and scaled Brier score (which is very similar to the Pearson correlation coefficient R²), the area under the receiver operating characteristic curve (AUROC), and the Hosmer-Lemshow goodness-of-fit p-value (HL).

Brier scores were consistently below the non-informative level of 0.25 and AUROC significantly higher than the non-informative level of 0.5 for the adjusted PPS, Karnofsky and ECOG models (table 1). The HL p-value was consistently greater than 0.1 only for PPS. SUPPORT and DEALE models did not predict fit our data well for survival at day one and month one, two and six. The AUROC takes a value close to 0.5, even though the Brier scores were relatively low and HL p-value greater than 0.05, this value is significantly close to 0.5 for SUPPORT and DEALE models (table 1).

View large

View Large

None of the prognostication models accurately predicated survival among our cohort of terminally ill patients. However, PPS consistently performed best in predicting survival in terminally ill patients followed by Karnofsky and ECOG.

No relevant conflicts of interest to declare.