Data quality in the study of the mechanisms of extracorporeal photochemotherapy

To the editor:

A recently published article by Berger et al describes studies in which the authors concluded that extracorporeal photopheresis (ECP) induces a subset of monocytes to differentiate into dendritic cells (DCs).¹  The core data are shown in Figures 1 and 2 of their paper, which depict aggregated flow cytometric analyses of blood from experimental groups, with representative histograms of monocytes stained for various markers; the proportion of cells expressing CD83 and human leukocyte antigen (HLA)–DR indicate differentiation to DCs.

Unfortunately, the data presented preclude drawing this conclusion because the histograms shown as examples in Figure 2 are improperly compensated for fluorescence spectral overlap. Monocytes express high levels of HLA-DR and, upon exposure to ECP, HLA-DR expression increased, resulting in signal levels that ranged from the top decade of fluorescence to off the scale. The data were undercompensated for spillover of HLA-DR fluorescence into the CD83 measurement channel, consequently, bright HLA-DR–expressing cells were incorrectly interpreted as CD83⁺. (It also appears that there was overcompensation of CD86 fluorescence into the CD80 channel, possibly resulting in some CD80⁺CD86⁺ cells being incorrectly interpreted as CD86⁻.)

The undercompensation is evident by the distribution skewing upward at a 45° angle at high HLA-DR expression levels. Our Figure 1 provides an example of this with data we collected, using either correct or incorrect compensation settings. Incorrect compensation often arises from use of compensation controls that are not as bright as the sample.²  Notably, undercompensation dramatically affects the ability to discern coexpression of markers, such as HLA-DR and CD83, or CD80 and CD86, as shown in the paper.

Even with proper compensation, there are still issues that preclude the use of “quadrant” gates to determine coexpression. Intrinsic measurement errors result in spreading of data with high measurement values.^2,3  The proper way to set gates that distinguish positive from negative expression is to use “fluorescence minus one” (FMO) controls,⁴  which are cells stained with all reagents except one. In this case, the appropriate discrimination of CD83⁺ from CD83⁻ cells would require a sample stained with all reagents except CD83; this distribution can then be used to define a discriminating gate, which might not be a straight line.² 

Finally, it is important to recognize that events appearing “off-scale” (at the extreme edge of the measurement range) cannot be properly compensated and, thus, must be excluded from such analyses.

Therefore, based on the data shown by Berger et al, conclusions cannot be reached regarding the proportion of double-positive cells. In addition, conclusions about mean fluorescence (as shown in Figure 3 of Berger et al) cannot be substantiated.

Proper implementation of these experiments could be achieved by (1) lowering the detector sensitivity to bring all events on-scale, using dimmer anti–HLA-DR reagents, or using anti–HLA-DR reagents at a suboptimal titer to reduce fluorescence; (2) using compensation controls that are at least as bright as the stained cells in the experiment; and (3) using appropriate FMO controls to set analysis gates. Only then can conclusions about the dim expression of CD83 by these cells be made.

The issue of miscompensation is prevalent in flow cytometric analyses. Indeed, flow cytometry is a technology that encompasses a wide range of expertise; complex and subtle artefacts can arise in many different areas of an experiment. It is understandable that researchers may not be sufficiently versed in the nuances to recognize problems. Nonetheless, it is a surprise that the researchers, their collaborators, and the reviewers did not detect this issue.

The fact that this happened highlights the need for journals to consult with flow cytometry experts on publications that rely heavily on cytometric analyses to ensure there are no serious concerns. The New England Journal of Medicine submits every nominally accepted paper for outside evaluation of statistics, because of the complex and subtle issues underlying many statistical analyses. We propose that a similar process should be undertaken by journals with regard to flow cytometric data.