Figure 1.
Comparison of CWS index across 4 different guidelines panels. Use of the CWS index to measure the expertise in guidelines panels. CWS is defined as a ratio between discrimination (which refers to the expert's differential evaluation of the various stimuli within a given set) and consistency (which refers to the judge's evaluation of the same stimuli over time or inconsistency, which is used in the formula to represent its complement). Higher the CWS ratio, the greater the expert's performance.14,17-20 This means that expertise is relative to one's peers. It is, therefore, challenging, if not impossible, to identify individuals in advance of a guideline panel without a "ground truth" or gold standard to compare these potential experts' knowledge.14,17-20 The participants were asked to make their judgments about the strength of recommendations (SoR) before vs immediately after the meeting deliberation. Although the CWS index for panel 2 was statistically significantly higher than other panels (P = .019), no statistically significant difference was detected in the judgments among the panelists’ members according to their role on the panel. (The analysis was performed using a linear mixed-effect model to control for judgments among panelists clustered within the guidelines panels). n = 45, based on unpublished data using ASH and other society guidelines. The figure shows a large variation in judgments among panel members making an argument for using the CWS index to identify the best people to serve on the panel. The results also demonstrate that judgments among the panels significantly differ as though they have relied on different types of knowledge.2,3 However, in this analysis, no difference was detected in judgments among the panelists’ members according to their role on the panel, possibly because of the small number of panelists (n = 45). The limitations of this system include the absence of an absolute value or cut-off, above which we label someone as an expert. In addition, as noted in the main text, the system cannot be applied a priori, making its practical application difficult.

Comparison of CWS index across 4 different guidelines panels. Use of the CWS index to measure the expertise in guidelines panels. CWS is defined as a ratio between discrimination (which refers to the expert's differential evaluation of the various stimuli within a given set) and consistency (which refers to the judge's evaluation of the same stimuli over time or inconsistency, which is used in the formula to represent its complement). Higher the CWS ratio, the greater the expert's performance.14,17-20 This means that expertise is relative to one's peers. It is, therefore, challenging, if not impossible, to identify individuals in advance of a guideline panel without a "ground truth" or gold standard to compare these potential experts' knowledge.14,17-20 The participants were asked to make their judgments about the strength of recommendations (SoR) before vs immediately after the meeting deliberation. Although the CWS index for panel 2 was statistically significantly higher than other panels (P = .019), no statistically significant difference was detected in the judgments among the panelists’ members according to their role on the panel. (The analysis was performed using a linear mixed-effect model to control for judgments among panelists clustered within the guidelines panels). n = 45, based on unpublished data using ASH and other society guidelines. The figure shows a large variation in judgments among panel members making an argument for using the CWS index to identify the best people to serve on the panel. The results also demonstrate that judgments among the panels significantly differ as though they have relied on different types of knowledge.2,3 However, in this analysis, no difference was detected in judgments among the panelists’ members according to their role on the panel, possibly because of the small number of panelists (n = 45). The limitations of this system include the absence of an absolute value or cut-off, above which we label someone as an expert. In addition, as noted in the main text, the system cannot be applied a priori, making its practical application difficult.

Close Modal
This Feature Is Available To Subscribers Only

or Create an Account

Close Modal
Close Modal