• Novel mixed methods were used to select risk factors for venous thromboembolism and bleeding RAMs in medical inpatients.

  • Risk factors were identified that require further research to confirm or refute their importance in RAMs.

Risk assessment models (RAMs) for venous thromboembolism (VTE) and bleeding in hospitalized medical patients inform appropriate use of thromboprophylaxis. Our aim was to use a novel approach for selecting risk factors for VTE and bleeding to be included in RAMs. First, we used the results of a systematic review of all candidate factors. Second, we used the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach to assess the certainty of the evidence for the identified factors. Third, we using a structured approach to select factors to develop the RAMs, by building on clinical and methodological expertise. The expert panel made judgments on whether to include, potentially include, or exclude risk factors, according to domains of the GRADE approach and the Delphi method. The VTE RAM included age >60 years, previous VTE, acute infections, immobility, acute paresis, active malignancy, critical illness, and known thrombophilia. The bleeding RAM included age ≥65 years, renal failure, thrombocytopenia, active gastroduodenal ulcers, hepatic disease, recent bleeding, and critical illness. We identified acute infection as a factor that was not considered in widely used RAMs. Also, we identified factors that require further research to confirm or refute their importance in a VTE RAM (eg, D-dimer). We excluded autoimmune disease which is included in the IMPROVE (International Medical Prevention Registry on Venous Thromboembolism) bleeding RAM. Our results also suggest that sex, malignancy, and use of central venous catheters (factors in the IMPROVE bleeding RAM) require further research. In conclusion, our study presents a novel approach to systematically identifying and assessing risk factors to be included or further explored during RAM development.

Venous thromboembolism (VTE), comprising deep venous thrombosis (DVT) and pulmonary embolism (PE), is a common cause of morbidity and mortality in hospitalized medical patients. The risk of VTE with hospitalization may increase more than eightfold.1,2  Patient- and disease-specific risk factors and their interaction modulate the magnitude and duration of the VTE risk in hospitalized medical patients.2  Patients receive thromboprophylaxis to mitigate this risk, but inappropriate use of VTE prophylaxis in low-risk patients may not meaningfully reduce VTE rates and may cause bleeding.2  Knowing a patient’s risk of VTE or bleeding would aid health care providers in selecting the appropriate prevention and management options to optimize patient care.3,4  Risk assessment models (RAMs) can help with stratification of an individual patients’ risk of developing a VTE or bleeding event and the choice of preventive measures.

The American Society of Hematology (ASH) guidelines for management of venous thromboembolism: prophylaxis for hospitalized and nonhospitalized medical patients described 15 existing RAMs, and the authors called for more research to improve and validate them.5-7  Derived from various studies, most of the identified RAMs were not developed based on a systematic review. This means that, although they are derived from large cohort studies, unmeasured potential risk factors in a specific cohort would have no possibility of being included in a RAM, whereas they could be captured as a candidate risk factor in a systematic review. We identified only 1 systematic review conducted 11 years ago that assessed VTE as an outcome in medical patients, but it did not include a meta-analysis or weighted statistical analysis of the prognostic factors.8 

There are several additional reasons for improving or validating currently existing RAMs. First, RAMs of single studies often rely on statistical significance, and in this setting, random variation or lack of power may or may not lead to the statistical significance of a risk factor in a prediction model. Second, statistical methods should be complemented by clinical expertise, to identify risk factors that are meaningful for health professionals. For example, if a RAM provides inaccurate or poorly calibrated estimates of VTE risk (ie, it over- or underpredicts by ignoring clinical context), it may mislead health care professionals. Third, the lack of using standardized definitions for risk factors causes confusion across RAMs. For example, Ye et al1  examined the various definitions of immobility used in previous studies and observed inconsistencies, which makes reproduction and validation of previous studies challenging.1  Fourth, RAMs should be able to predict specific events accurately and still be relatively easy to use. Fifth, there is no universal consensus on use of a specific RAM in hospitalized medical patients, in part because of the reasons just mentioned.9 

Therefore, we used a novel approach to support the development of new RAMs and inform the updating of widely used RAMs for VTE and bleeding in hospitalized acutely, critically, and chronically ill medical patients. We first conducted a systematic review of all relevant risk factors in hospitalized medical patients.10  In tandem, we used extensive clinical and methodological expertise to assess the certainty in the identified risk factors and select them by using a structured approach that requires clinical expertise.

Ethics

After the review of the project, the Hamilton Integrated Research Ethics Board waived the need for ethics approval for this study.

Study design

We conducted a study that combined systematic review methods and an assessment of the certainty of the evidence according to Grading of Recommendations Assessment, Development, and Evaluation (GRADE). This work then informed a structured Delphi-based expert judgment to include, potentially include, or exclude risk factors for VTE and bleeding in hospitalized medical patients, by using the GRADE Evidence to Decision (EtD) criteria.11  The process is described in Figure 1.

Figure 1.

Flow chart of our approach to develop risk assessment models.

Figure 1.

Flow chart of our approach to develop risk assessment models.

Close modal

Participants

The expert panel included clinicians and researchers with expertise in management of VTE and bleeding in hospitalized medical patients, and in the development, validation, and implementation of RAMs for clinical practice. Panel members participated in a face-to-face panel meeting, responded to surveys and questionnaires, and provided feedback on reports. They completed declaration-of-interest forms to ensure transparency on potentially existing conflicts with regard to existing RAMs and other factors.

The research team selected members of the expert panel by using purposive sampling and the following criteria:

  • Leading author on a journal article on VTE risk assessment in hospitalized medical patients

  • 2015 Centers for Disease Control (CDC) health care–associated VTE Challenge Champion leader12 

  • Hospital VTE “champion”

  • ASH Prevention of VTE Nonsurgical Patients Guideline author7 

  • International representation

The research team compiled the evidence for presentation, drafted the questions for the Delphi process, analyzed the responses, and summarized the results.

Systematic review

Before this study, we conducted a systematic review to identify all potential risk factors for VTE and bleeding in hospitalized medical patients, which we have described in detail elsewhere.10  In brief, we searched Medline and EMBASE from inception to May 2018. We considered prognostic factors and RAM studies that identified potential prognostic factors for the outcomes VTE and bleeding in hospitalized adult acutely, critically, or chronically ill medical patients. We defined VTE as any symptomatic or asymptomatic DVT or PE within 90 days after discharge. Bleeding included major or nonmajor but clinically significant bleeding within 90 days after discharge.13,14  Reviewers extracted data in duplicate and independently and assessed the certainty of the evidence by using the GRADE approach.15  The results of the systematic review were used for this study.10 

Delphi process

Delphi round 1.

We asked the expert panel by e-mail to provide input (eg, identify gaps) on the list of risk factors that we identified through the systematic review. Expert panelists responded confidentially and independently so as not to influence other panel members. Subsequent Delphi rounds aimed to build on this by requesting that the experts make clinical and methodological judgements based on the available evidence.15,16 

EtD frameworks.

For the second and third round of the Delphi process, we used the GRADEpro Guideline Development Tool to facilitate the process.17  The tool includes standardized tables and frameworks:

  1. Evidence profiles. During the third round of the Delphi process, we used these tables, which provide synthesized evidence for each risk factor based on a meta-analysis and an assessment of the certainty of the evidence rated as high, moderate, low, or very low. We based these ratings on considerations of risk of bias, indirectness, inconsistency, and imprecision.15 

  2. GRADE EtD frameworks. During the second and third rounds of the Delphi process, we used the following criteria from the EtDs to facilitate the panel’s decision-making process18 : benefits and harms, resource requirements, equity, acceptability, and feasibility. These criteria allowed the expert panel to make judgements about the risk factors that were both evidence-based and clinically relevant. Although this approach followed good practice in RAM development that suggests attaining high predictability while remaining relatively simple and applicable in clinical settings, it is novel, as it uses a structured approach based on EtD criteria.18,19 

Delphi round 2.

We held a face-to-face panel meeting to discuss the systematic review findings and the approach to judge which risk factors should be included in the RAM for VTE and bleeding. We presented the results of the systematic review and the results of the first round of the Delphi process. The results included individual and pooled estimates of the associations and the corresponding confidence intervals for each identified risk factor using forest plots for each meta-analysis. After reviewing the results and discussing resource requirements, equity, feasibility, and acceptability, the panel was asked to categorize the risk factors into 3 sets: included, potentially included, and excluded. We defined the included risk factors as those that should be included in a RAM. We defined the potentially included factors as candidates for consideration in a RAM. The excluded risk factors were those that should not be considered for a RAM. Reasons for exclusion included potential interaction with other factors or no association with the outcome.

The research team and the expert panel noted inconsistencies in the definitions of the risk factors across studies and agreed that standardizing the definitions is critical for deciding which ones to include in a RAM and for future research, data collection purposes, and most important, clinical relevance. We standardized the definitions of the included and potentially included risk factors by reviewing the definitions of the original study as detailed in supplemental Tables 6 and 7. We then obtained input from the panel to draft the suggested definitions.

Delphi round 3.

We conducted a Web-based anonymous survey through SurveyMonkey.20  We asked questions after presenting results of the previous round, the assessment of the certainty of the evidence, and a descriptive summary of findings for each risk factor from the systematic review. In this round, we asked the panel members to formally judge the effect estimate of the meta-analysis, the resource requirements, and the impact on equity, acceptability, and feasibility of each risk factor. Based on these criteria, the panel judged whether these risk factors should be included, potentially included, or excluded.

We determined a priori that we would make final judgments based on simple majority votes. However, when votes were spread across all 3 categories and both the included and excluded category had more than 1 vote each, we determined that the risk factor should be potentially included.

To harmonize the definitions of risk factors, we shared draft definitions with the expert panel based on the literature we identified. We asked them to review the information and provide feedback, which we incorporated in final definitions of the risk factors (supplemental Table 8).

Final risk RAMs.

The risk of VTE or bleeding for each risk factor was presented as an odds ratio (OR) with the relative 95% confidence interval (derived from the meta-analysis). To develop the VTE and bleeding RAMs, we log transformed the ORs into β coefficients and determined the linear predictor (Y) for VTE or bleeding.21  The final RAMs, presented as regression models, are given as21LP(Y)=β0+β1X1+β2X2, where LP is the linear predictor Y of the outcome VTE or bleeding that is derived from the logistic regression model, where β0 is the intercept, β1 is the β coefficient for the first risk factor, and X1 is the first risk factor, and so on.21  To determine the contribution of each risk factor to the overall risk of VTE or bleeding, we summed the β coefficients, divided each by the total, and multiplied by 100 (Tables 2 and 3). We did not evaluate nonlinear logistic regression models.

We then presented a case scenario where we computed the patient-specific risk or predicted probability of VTE and bleeding based on the formula P = odds/(odds + 1), where odds is the exponent of the linear predictor (Y) (odds = ey), and Y is the linear predictor that was derived from the final RAM. If the risk factor in the RAM is present, X is given a value of 1 and if absent, a value of 0. In the case scenario, we applied a calculated β0 corresponding to a VTE or bleeding risk of 0% because of the assumption that we would identify all risk factors. Absence of any risk factor would therefore approach a risk of 0%.

Response rate and participant characteristics

We included 9 physicians who were both clinical and methodological experts in the field (M.C., M.K.G., H.J.S., F.A.S., A.C.S., M.B.S., S.C.W., N.A.Z., and L.M.), one of whom was a biostatistician (L.M.)22,23  (supplemental Table 1). For each of the 3 rounds of the Delphi process, we achieved a 100% response rate.

Systematic review findings

We identified 17 eligible studies, 14 of which reported on VTE and described 29 candidate prognostic factors,24-37  and 3 studies that reported on bleeding and described 17 candidate factors.38-40  Supplemental Tables 2 and 3 provide the evidence profiles for VTE- and bleeding-related prognostic factors. A detailed description of the results of the systematic review and corresponding forest plots of the meta-analyses are published elsewhere.10 

VTE model risk factor selection

Delphi round 1.

Listed in Table 1 are the 29 potential risk factors identified from the systematic review and the additional 3 factors suggested by the expert panel, for a total of 32. The risk factors were all evaluated at the time of admission, except for immobility, acute paresis, and critical illness, which were assessed both at admission and during the hospital stay.

Delphi round 2.

The panel judged 9 of the 32 factors to be included: age, previous VTE, acute infections, immobility, acute paresis, malignancy, critical illness, D-dimer level, and known thrombophilia. They potentially included the following 14 factors: sex, respiratory failure, severe stroke, autoimmune disease, obesity, thrombocytosis, central venous catheter (CVC) use, leg edema, fever, heart rate, leukocytosis, recent long bone fractures, recent travel, and history of heart failure (acute heart failure was excluded). The expert panel excluded renal failure, hormone use, tobacco, and coronary artery disease, as the results showed little to no association with the outcome VTE. They excluded varicose veins, low Barthel index score, elevated C-reactive protein, and fibrinogen levels, because they perceived these risk factors to be nonspecific and not routinely measured in patients admitted to the hospital. They also excluded recent surgery <30 days before admission, so that the focus would be on medical patients.

Delphi round 3.

Based on our voting criteria, the panel agreed to include the following 8 risk factors for the VTE RAM: age >60 years (by consensus), previous VTE (by consensus), acute infections, immobility, acute paresis, active malignancy (by consensus, a history of malignancy was excluded), critical illness, and known history of thrombophilia. The final round of our approach eliminated sex, elevated heart rate, and recent air travel from the list of potentially included risk factors (Table 2). The judgements for these factors are detailed in supplemental Table 3. The panel also agreed on definitions for these risk factors as described in supplemental Table 8.

Bleeding model risk factor selection

Delphi round 1.

A total of 17 risk factors were candidates, based on the systematic review (Table 1). The expert panel agreed and did not suggest any additional factors. The risk factors were all evaluated at the time of admission, except for use of antithrombotic medication and rehospitalization, which were assessed after the index admission.

Delphi round 2.

During the face-to-face meeting, the panel judged 8 of the 17 factors to be included in the RAM: age ≥65 years, critical illness, thrombocytopenia, active gastroduodenal ulcer in the past 3 months, hepatic disease, recent bleeding, blood dyscrasias, and use of antithrombotic medication. They agreed that 5 of the 17 factors were candidates for inclusion: sex, renal failure, malignancy, autoimmune disease, and CVC use. The expert panel excluded hormone use, because the results showed little to no association with the outcome bleeding. They excluded obesity because of the low certainty of evidence that it is a risk factor. They also excluded low hemoglobin and rehospitalization because of the very low certainty of evidence and the lack of specificity with the outcome bleeding.

Delphi round 3.

The panel determined that the following risk factors should be included in the bleeding RAM: age ≥65 years, renal failure, thrombocytopenia (by consensus), active gastroduodenal ulcer in the past 3 months, hepatic disease, recent bleeding, use of antithrombotic medication, and critical illness. However, we opted not to include antithrombotic medication use as a risk factor in the bleeding RAMs, because our aim was to develop a RAM that would assist health care professionals in identifying medical patients at admission who may be at increased risk of bleeding caused by anticoagulants, to appropriately weigh benefits and harms before starting treatment. Sex, CVC use, blood dyscrasias and malignancy were rated as potentially included (Table 2) but autoimmune disease was excluded. The judgments for these factors are detailed in supplemental Table 5. Supplemental Table 8 presents the agreed upon definitions for these risk factors.

Final RAMs.

We developed the VTE and bleeding RAMs by using the included risk factors detailed in Table 3. Based on these risk factors, their corresponding β coefficients, and a baseline VTE or bleeding risk of 0%, the regression models for the linear predictors VTE and bleeding, respectively, are:

The risk factors with the largest contributions to the overall VTE risk (Table 3) are previous VTE (22.6%), known thrombophilia (22.2%), immobility (14.5%), and acute paresis (13.6%), whereas those with the largest contributions to the overall bleeding risk (Table 3) are recent bleeding (30.2%), active gastroduodenal ulcers in past 3 months (18.6%), and critical illness (13.7%).

Case scenario.

This case scenario suggests how these RAMs can be used in clinical practice. Consider a young, acutely ill medical patient (XAge >60 = 0) who is admitted to the hospital for an acute infection (XAcute infection = 1). The patient has no history of VTE (XPrevious VTE = 0), is mobile (XImmobility = 0), has no acute paresis (XParesis = 0), no active malignancy (XActive Malignancy = 0), no critical illness (XCritical illness = 0), and no known history of thrombophilia (XKnown history of thrombophilia = 0). Based on the VTE RAM, this patient has a linear predictor (y) of −5.41.

The following formula was used to calculate the odds:

Odds = ey = e−5.41 = 0.0045,

and this formula for calculating the probability:
Therefore, the patient has a VTE probability of ∼0.45%.

In terms of the bleeding risk profile, the patient (XAge ≥65 = 0) reported having known thrombocytopenia (XThrombocytopenia = 1) and an active gastroduodenal ulcer in the past 3 months (XActive gastroduodenal ulcers = 1), but had none of the other risk factors included in the bleeding RAM; that is, the patient had no recent bleeding episode (XRecent bleeding = 0), impaired renal function (XRenal failure = 0), hepatic disease (XHepatic disease = 0), or any critical illness (XCritical illness = 0). Based on the developed bleeding RAM, this patient has a linear predictor (y) of −4.01 which corresponds to a bleeding probability of ∼1.8% based on the formulas.

One recommendation in the ASH guidelines assessed the effect of any parenteral anticoagulation (unfractionated heparin, low-molecular-weight heparin, or fondaparinux) compared with none.7  Based on the results of the meta-analyses, the relative risk was 0.58 for combined symptomatic PE and DVT and 1.48 for major bleeding.7  Based on the predicted probabilities in the case scenario and on the effects of parenteral anticoagulation on VTE and bleeding, if the patient were prescribed thromboprophylaxis, the absolute risk of VTE would be reduced by ∼0.2%, whereas the absolute risk of bleeding would increase by ∼0.9%, amounting to an absolute risk for VTE of 0.26% and for bleeding of 2.66%.

These risk estimates are useful for implementing the corresponding ASH recommendations regarding acutely or critically ill medical patients: mechanical VTE prophylaxis compared with a combination of pharmacological and mechanical or pharmacological VTE prophylaxis alone.7  Given the bleeding risk and if the patient places a relatively high value on avoiding bleeding complications, the harms would outweigh the benefits. Interpreting the conditional recommendation, “In acutely or critically ill medical patients, the ASH guideline panel suggests using pharmacological VTE prophylaxis over mechanical VTE prophylaxis (conditional recommendation, very low certainty in the evidence of effects.),”7  suggests not using pharmacological thromboprophylaxis for this patient. Thus, the following ASH recommendation should be used for this patient: “In acutely or critically ill medical patients who do not receive pharmacological VTE prophylaxis, the ASH guideline panel suggests [emphasis in the original] using mechanical VTE prophylaxis over no VTE prophylaxis (conditional recommendation, moderate certainty in the evidence of effects).”7 

Summary of findings

We used a novel approach to systematically identify and assess risk factors to support the development of a RAM and inform the update of widely used RAMs for VTE and bleeding in hospitalized acutely, critically, or chronically ill medical patients. First, we conducted a systematic review of all relevant risk factors in hospitalized medical patients.10  Second, we assessed the certainty of the evidence in identified risk factors. Third, we selected the factors to include in the RAMs, using an innovative structured approach based on GRADE that required extensive clinical and methodological expertise. The expert panel made judgments on whether to include, potentially include, or exclude identified risk factors from the final RAMs using the Delphi method based on GRADE criteria. This novel approach allowed us to identify risk factors that should be included in a RAM and factors that require further exploration. These findings support the development or update of a RAM that can accurately predict specific events while remaining relatively simple and applicable to use in clinical settings. If a RAM provides inaccurate over- or underestimates of future event occurrences, it may lead to mismanagement of patient care and health care resources. On the other hand, if a model has high predictability power but is difficult to apply, time consuming, costly, or less relevant, it will not be commonly used. However, these RAMs should be externally validated and should be assessed in prospective impact studies.

Our 8-factor VTE RAM was developed based on risk factors assessed at admission and during hospital stay and includes age >60 years, previous VTE, acute infections, immobility, acute paresis, active malignancy, critical illness, and known history of thrombophilia. Our 7-factor bleeding RAM was developed based on risk factors assessed only at admission and includes age ≥65 years, renal failure, thrombocytopenia, active gastroduodenal ulcer in the past 3 months, hepatic diseases, recent bleeding, and critical illness. The potentially included risk factors (Table 2) require further study to confirm or refute their importance for the respective RAMs.

Comparison with other RAMs

We developed RAMs for VTE and bleeding in hospitalized medical patients that were similar but not identical to some widely used RAMs in current practice, such as the IMPROVE (International Medical Prevention Registry on Venous Thromboembolism) VTE RAM, IMPROVE Bleeding RAM, Intermountain RAM, MITH (Medical Inpatients and Thrombosis) RAM, and PADUA VTE RAM.33,35,38,41,42  Compared with the IMPROVE VTE RAM, acute infection is an additional important risk factor.33  We identified 12 additional candidate risk factors, 5 of which were considered in other VTE RAMs, including CVC use in the Intermountain RAM,42  respiratory failure and heart failure in the PADUA and MITH VTE RAMs, severe stroke in the PADUA VTE RAM, and thrombocytosis and leukocytosis in the MITH RAM.35,41  Compared with the PADUA model, we found that coronary artery disease does not predict VTE risk, and obesity requires further research.41  As for risk of bleeding, our RAM suggests that autoimmune disease did not predict bleeding risk because of conflicting results in the included studies.38  Also, we did not include sex, malignancy, and CVC use, which are included in the IMPROVE Bleeding RAM, because we judged that these were candidate risk factors that require further exploration.38 

Strengths

Our study is based on rigorous methods that are innovative for several reasons. First, the systematic review conducted by our group and the input from the expert panel in comprehensively identifying all potential risk factors is a limitation of cohort studies. Second, assessing the certainty of the evidence based on a structured framework allows for an expression of the confidence in the predictive ability of the factors. Third, the expert panel made judgements on the inclusion of risk factors in the RAMs using GRADE criteria by accounting for their resource requirements, impact on equity, acceptability, and feasibility, all of which are relevant in clinical practice. Fourth, we are not aware of prior use of this approach to developing RAMs. Fifth, we standardized the definitions of the included and potentially included risk factors to decrease variability in methods of measurement across settings. Standardized definitions will provide more clarity to health care professionals, including researchers, when evaluating and weighing patients’ risks of VTE and bleeding and subsequent management options. Our work also strongly suggests the need to standardize definitions of risk factors if we are to make further progress in this area.

Limitations

Potential limitations of the systematic review findings are the inconsistency and variability across eligibility criteria in the included studies and variability in study design, study type, sample size, and definitions of the risk factors. Other limitations included the inconsistency in the diagnostic approaches used across studies and the contamination of the study population with nonmedical hospitalized patients for some of the risk factors. We were unable to conduct a meta-regression to adjust for study level characteristics, because the number of studies was too small for this analysis. Also, we did not conduct an external validation which is an essential next step. However, validation is a continuous process, and our approach should be viewed as a method of validating the content of current widely used RAMs. For example, our VTE RAM validated the findings in the IMPROVE VTE RAM, highlighted the need to include acute infection as a factor, and provided a list of candidate factors that require further evaluation.

Implications for practice

The findings of this study can aid in RAM development or updating. Our findings can also help health care professionals in evaluating the risk of VTE from index admission until discharge and the risk of bleeding on admission in hospitalized medical patients for optimal patient management. Ideally, this can be achieved by integrating the RAMs in clinical decision aids to assist with deriving individual-based recommendations from published population-based guideline recommendations for shared decision making.

Implications for future research

Our developed RAMs should be tested in an external validation study using individual patient data sets. Such validation is essential before conducting an impact analysis that would allow the RAMs to be adopted in routine clinical practice. Also, the potentially included risk factors should be explored in further research. We standardized the definitions of the risk factors to help researchers build more uniform datasets and registries. Decreasing variability will facilitate the reproduction and validation of studies across settings.

After review of the work and discussions with the expert panel, we noted that a risk assessment for VTE and bleeding conducted only on admission is insufficient and will not account for a change in risk factors throughout hospitalization. For example, transferring a patient from a medical ward to the intensive care unit or development of acute renal failure or acute infection may change the risk level. Therefore, developing a system for dynamic risk assessment of hospitalized medical patients from admission to discharge is important. The shortening of hospital lengths of stay, the lack of routine postdischarge thromboprophylaxis, and the recent availability of thromboprophylactic agents that can be used for extended thromboprophylaxis in hospitalized medical patients makes testing our proposed VTE RAM for dynamic use especially important.

Conclusion

We developed a novel structured approach for selecting risk factors for VTE and bleeding in hospitalized medical patients that is evidence based, clinically meaningful, and relevant. Our findings support the development of new RAMs and the update of widely used RAMs. Also, our findings inform external validation and prospective impact assessment studies that may be undertaken to evaluate the performance of these RAMs in assessing VTE and bleeding risk for this population. These findings may assist decision makers in weighing the risk of VTE with that of bleeding, to appropriately select VTE prevention strategies and optimize patient care for different patient risk groups.

Original data are available by e-mail request to the corresponding author.

The authors thank ASH for providing the space to hold the panel workshop and for logistical support with the preparations.

This work was supported by a subcontract (200-2016-92458) from the US Centers for Disease Control and Prevention (CDC) through Karna LLC.

The findings and conclusions of this report are those of the authors and do not necessarily represent the official position of the CDC.

Contribution: A.J.D., L.M., M.R., and H.J.S. analyzed the data; A.J.D. and H.J.S. drafted the manuscript; and all authors interpreted the results, critically revised the manuscript, and approved of the final version.

Conflict-of-interest disclosure: A.J.D., I.E.-I., A.I., E.A.A., and H.J.S. are members of the GRADE working group. H.J.S. is cochair of the panel that developed the American Society of Hematology 2018 guidelines for management of venous thromboembolism: prophylaxis for hospitalized and nonhospitalized medical patients (ASH 2018 VTE guidelines) and obtained grant funding from the CDC for this study. A.C.S. has received remuneration for consulting work for Bayer, Janssen, and Portola; has obtained research support grants from Boehringer Ingelheim, Janssen, and the Centers for Medicare and Medicaid Services; and reports an intellectual conflict as the leader of the group that developed and validated the IMPROVE VTE tool for VTE risk assessment in hospitalized medical patients. M.B.S. has received remuneration for consulting work for Bayer, Janssen, Pfizer, and Portola and research support grants from Boehringer-Ingelheim, Janssen, Portola, and Roche. M.C. is a former member of the board of directors (2013-2017) of the American Heart Association and chaired the panel that developed the ASH 2018 VTE guidelines. F.A.S. and N.A.Z. participated as panel members for the ASH 2018 VTE guidelines. N.A.Z. reported receiving honoraria in 2017 from ASH for the Highlights of the ASH 2017 meeting (Dallas, New York, and Latin America). S.C.W. served as cochair of the panel for Antithrombotic Therapy for VTE Disease. N.A.Z. and M.C. report intellectual conflicts as leaders of the group that derived and validated the MITH RAM for VTE risk assessment in hospitalized medical patients. The remaining authors declare no competing financial interests.

Correspondence: Holger J. Schünemann, Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main St West, Hamilton, ON L8N 3Z5, Canada; e-mail: schuneh@mcmaster.ca.

1.
Ye
F
,
Bell
LN
,
Mazza
J
,
Lee
A
,
Yale
SH
.
Variation in Definitions of Immobility in Pharmacological Thromboprophylaxis Clinical Trials in Medical Inpatients: A Systematic Review
.
Clin Appl Thromb Hemost
.
2018
;
24
(
1
):
13
-
21
.
2.
Chan
NC
,
Gross
PL
,
Weitz
JI
.
Addressing the burden of hospital-related venous thromboembolism: the role of extended anticoagulant prophylaxis
.
J Thromb Haemost
.
2018
;
16
(
3
):
413
-
417
.
3.
Hemingway
H
,
Croft
P
,
Perel
P
, et al;
PROGRESS Group
.
Prognosis research strategy (PROGRESS) 1: a framework for researching clinical outcomes
.
BMJ
.
2013
;
346
:
e5595
.
4.
Steyerberg
EW
,
Moons
KG
,
van der Windt
DA
, et al;
PROGRESS Group
.
Prognosis Research Strategy (PROGRESS) 3: prognostic model research
.
PLoS Med
.
2013
;
10
(
2
):
e1001381
.
5.
Stuck
AK
,
Spirk
D
,
Schaudt
J
,
Kucher
N
.
Risk assessment models for venous thromboembolism in acutely ill medical patients. A systematic review
.
Thromb Haemost
.
2017
;
117
(
4
):
801
-
808
.
6.
Huang
W
,
Anderson
FA
,
Spencer
FA
,
Gallus
A
,
Goldberg
RJ
.
Risk-assessment models for predicting venous thromboembolism among hospitalized non-surgical patients: a systematic review
.
J Thromb Thrombolysis
.
2013
;
35
(
1
):
67
-
80
.
7.
Schünemann
HJ
,
Cushman
M
,
Burnett
AE
, et al
.
American Society of Hematology 2018 guidelines for management of venous thromboembolism: prophylaxis for hospitalized and nonhospitalized medical patients
.
Blood Adv
.
2018
;
2
(
22
):
3198
-
3225
.
8.
Rocha
AT
,
Paiva
EF
,
Lichtenstein
A
,
Milani
R
Jr.
,
Cavalheiro
CF
,
Maffei
FH
.
Risk-assessment algorithm and recommendations for venous thromboembolism prophylaxis in medical patients
.
Vasc Health Risk Manag
.
2007
;
3
(
4
):
533
-
553
.
9.
Cruden
P
,
Cushman
M
,
Repp
AB
.
Hospitalist assessment of venous thromboembolism and bleeding risk: a survey study
.
Thromb Res
.
2019
;
178
:
155
-
158
.
10.
Darzi
AJ
,
Karam
SG
,
Charide
R
, et al
.
Prognostic factors for VTE and bleeding in hospitalized medical patients: a systematic review and meta-analysis
.
Blood
.
2020
;
135
(
20
):
1788
-
1810
.
11.
Colonna
P
,
Andreotti
F
,
Ageno
W
,
Pengo
V
,
Marchionni
N
.
Clinical conundrums in antithrombotic therapy management: a Delphi Consensus panel
.
Int J Cardiol
.
2017
;
249
:
249
-
256
.
12.
Centers for Disease Control and Prevention
. The 2015 Healthcare-Associated Venous Thromboembolism Prevention Challenge Champions.
2020
. https://www.cdc.gov/ncbddd/dvt/ha-vte-challenge.html. Accessed 29 January 2020.
13.
Kaatz
S
,
Ahmad
D
,
Spyropoulos
AC
,
Schulman
S
;
Subcommittee on Control of Anticoagulation
.
Definition of clinically relevant non-major bleeding in studies of anticoagulants in atrial fibrillation and venous thromboembolic disease in non-surgical patients: communication from the SSC of the ISTH
.
J Thromb Haemost
.
2015
;
13
(
11
):
2119
-
2126
.
14.
Schulman
S
,
Kearon
C
;
Subcommittee on Control of Anticoagulation of the Scientific and Standardization Committee of the International Society on Thrombosis and Haemostasis
.
Definition of major bleeding in clinical investigations of antihemostatic medicinal products in non-surgical patients
.
J Thromb Haemost
.
2005
;
3
(
4
):
692
-
694
.
15.
Iorio
A
,
Spencer
FA
,
Falavigna
M
,
Alba
C
,
Lang
E
,
Burnand
B
, et al
.
Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients
.
BMJ
.
2015
;
350
:
h870
.
16.
Helmer
O
.
Analysis of the future: the Delphi method
.
Santa Monica, CA
:
The Rand Corporation
;
1967
.
17.
GRADEpro GDT
.
GRADEpro Guideline Development Tool.
2020
. www.gradepro.org. Accessed 29 January 2020.
18.
Alonso-Coello
P
,
Schünemann
HJ
,
Moberg
J
, et al
.
GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction
.
BMJ
.
2016
;
353
:
i2016
.
19.
Alonso-Coello
P
,
Oxman
AD
,
Moberg
J
, et al
. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines.
BMJ
.
2016
;
353
:
i2089
.
20.
SurveyMonkey
. Get answers with surveys 1999-2018. https://www.surveymonkey.com. Accessed 15 August 2018.
21.
Debray
TP
,
Koffijberg
H
,
Nieboer
D
,
Vergouwe
Y
,
Steyerberg
EW
,
Moons
KG
.
Meta-analysis and aggregation of multiple published prediction models
.
Stat Med
.
2014
;
33
(
14
):
2341
-
2362
.
22.
Fitch
K
,
Bernstein
SJ
,
Aguilar
MD
, et al
.
The RAND/UCLA appropriateness method user’s manual
.
Santa Monica, CA
:
The Rand Corporation
;
2001
.
23.
Hsu
C-C
,
Sandford
BA
. The Delphi Technique: Making Sense of Consensus. Semantic Scholar.
2007
https://www.semanticscholar.org/paper/The-Delphi-Technique%3A-Making-Sense-of-Consensus-Hsu-Sandford/edd0a804fccb2a7587f785bf255a52a61fc23284. Accessed 3 January 2020.
24.
Barclay
SM
,
Jeffres
MN
,
Nguyen
K
,
Nguyen
T
.
Evaluation of pharmacologic prophylaxis for venous thromboembolism in patients with chronic liver disease
.
Pharmacotherapy
.
2013
;
33
(
4
):
375
-
382
.
25.
Bembenek
J
,
Karlinski
M
,
Kobayashi
A
,
Czlonkowska
A
.
Early stroke-related deep venous thrombosis: risk factors and influence on outcome
.
J Thromb Thrombolysis
.
2011
;
32
(
1
):
96
-
102
.
26.
Fan
J
,
Li
X
,
Cheng
Y
,
Yao
C
,
Zhong
N
;
Investigators Group
.
Measurement of D-dimer as aid in risk evaluation of VTE in elderly patients hospitalized for acute illness: a prospective, multicenter study in China
.
Clin Invest Med
.
2011
;
34
(
2
):
E96
-
E104
.
27.
Grant
PJ
,
Greene
MT
,
Chopra
V
,
Bernstein
SJ
,
Hofer
TP
,
Flanders
SA
.
Assessing the caprini score for risk assessment of venous thromboembolism in hospitalized medical patients
.
Am J Med
.
2016
;
129
(
5
):
528
-
535
.
28.
Kelly
J
,
Rudd
A
,
Lewis
RR
,
Coshall
C
,
Moody
A
,
Hunt
BJ
.
Venous thromboembolism after acute ischemic stroke: a prospective study using magnetic resonance direct thrombus imaging
.
Stroke
.
2004
;
35
(
10
):
2320
-
2325
.
29.
Mahan
CE
,
Liu
Y
,
Turpie
AG
, et al
.
External validation of a risk assessment model for venous thromboembolism in the hospitalised acutely-ill medical patient (VTE-VALOURR)
.
Thromb Haemost
.
2014
;
112
(
4
):
692
-
699
.
30.
Ota
S
,
Yamada
N
,
Tsuji
A
,
Ishikura
K
,
Nakamura
M
,
Ito
M
.
Incidence and clinical predictors of deep vein thrombosis in patients hospitalized with heart failure in Japan
.
Circ J
.
2009
;
73
(
8
):
1513
-
1517
.
31.
Rosenberg
D
,
Eichorn
A
,
Alarcon
M
,
McCullagh
L
,
McGinn
T
,
Spyropoulos
AC
.
External validation of the risk assessment model of the International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) for medical patients in a tertiary health system
.
J Am Heart Assoc
.
2014
;
3
(
6
):
e001152
.
32.
Rothberg
MB
,
Lindenauer
PK
,
Lahti
M
,
Pekow
PS
,
Selker
HP
.
Risk factor model to predict venous thromboembolism in hospitalized medical patients
.
J Hosp Med
.
2011
;
6
(
4
):
202
-
209
.
33.
Spyropoulos
AC
,
Anderson
FA
Jr.
,
FitzGerald
G
, et al;
IMPROVE Investigators
.
Predictive and associative models to identify hospitalized medical patients at risk for VTE
.
Chest
.
2011
;
140
(
3
):
706
-
714
.
34.
Yi
X
,
Lin
J
,
Han
Z
,
Zhou
X
,
Wang
X
,
Lin
J
.
The incidence of venous thromboembolism following stroke and its risk factors in eastern China
.
J Thromb Thrombolysis
.
2012
;
34
(
2
):
269
-
275
.
35.
Zakai
NA
,
Callas
PW
,
Repp
AB
,
Cushman
M
.
Venous thrombosis risk assessment in medical inpatients: the medical inpatients and thrombosis (MITH) study
.
J Thromb Haemost
.
2013
;
11
(
4
):
634
-
641
.
36.
Zakai
NA
,
Wright
J
,
Cushman
M
.
Risk factors for venous thrombosis in medical inpatients: validation of a thrombosis risk score
.
J Thromb Haemost
.
2004
;
2
(
12
):
2156
-
2161
.
37.
Zhou
H
,
Hu
Y
,
Li
X
, et al
.
Assessment of the Risk of Venous Thromboembolism in Medical Inpatients using the Padua Prediction Score and Caprini Risk Assessment Model
.
J Atheroscler Thromb
.
2018
;
25
(
11
):
1091
-
1104
.
38.
Decousus
H
,
Tapson
VF
,
Bergmann
J-F
, et al;
IMPROVE Investigators
.
Factors at admission associated with bleeding risk in medical patients: findings from the IMPROVE investigators
.
Chest
.
2011
;
139
(
1
):
69
-
79
.
39.
Mahan
CE
,
Fisher
MD
,
Mills
RM
, et al
.
Thromboprophylaxis patterns, risk factors, and outcomes of care in the medically ill patient population
.
Thromb Res
.
2013
;
132
(
5
):
520
-
526
.
40.
Patell
R
,
Gutierrez
A
,
Rybicki
L
,
Khorana
AA
.
Identifying predictors for bleeding in hospitalized cancer patients: a cohort study
.
Thromb Res
.
2017
;
158
:
38
-
43
.
41.
Barbar
S
,
Noventa
F
,
Rossetto
V
, et al
.
A risk assessment model for the identification of hospitalized medical patients at risk for venous thromboembolism: the Padua Prediction Score
.
J Thromb Haemost
.
2010
;
8
(
11
):
2450
-
2457
.
42.
Woller
SC
,
Stevens
SM
,
Jones
JP
,
Lloyd
JF
,
Evans
RS
,
Aston
VT
, et al
.
Derivation and validation of a simple model to identify venous thromboembolism risk in medical patients
.
Am J Med
.
2011
;
124
(
10
):
947
-
954.e2
.

Author notes

The full-text version of this article contains a data supplement.

Supplemental data