Response: Making “perfect” the enemy of good

Response

We appreciate the interest of Ulirsch et al¹  in our recent publication describing distinctive transcriptional profiles observed in ex vivo–differentiated CD34⁺ cells from Diamond-Blackfan anemia (DBA) patients.²  Ulirsch et al highlight potential problems encountered when comparing diseased and normal conditions, problems that apply to all but the simplest of disease models, which raises a more philosophical question: in the quest for perfect controls, can any diseased cell ever be compared with any control? Ulirsch et al feel that these problems are confounding variables that can undermine confidence in results to the point where they can be discounted, specifically in the case of our results. Ulirsch et al do a commendable job of explaining the concept of confounding variables. We view their interpretations of our data as highly selective and far short of definitive. We believe a broader perspective is warranted.

The underlying pathophysiology of DBA at this time is not well understood. The Sankaran group has published a study analyzing the transcriptional profiles of bone marrow CD34⁺CD71^highCD45RA⁻ cells sorted from 3 DBA patients, 2 with RPS19 mutations and 1 of unknown genotype, with ribosomal protein (RP) mutations that suggested lower levels of GATA1 gene target expression compared with control cells.³  This observation was supported by introduction of short hairpin RNA (shRNA) vectors into normal CD34⁺ cells to knockdown either RPS19 or RPL11 as a model for RP-mediated DBA. Based on these results, they proposed that the ribosome deficiency in patients with RP mutations specifically causes defective translation of GATA1 messenger RNA leading to downstream erythroid failure.

We evaluated a larger and more diverse population of ex vivo–generated erythroid progenitor cells from DBA patients (defined as CD44⁺CD235⁻). Confronted by the need to test for expression differences between 3 groups across thousands of genes, we chose to adjust our significance levels by using a false discovery rate, with q stringently selected as 0.10. This controls the probability of erroneously labeling a difference as statistically significant; it does not assure that all genes with differential expression will be identified in our data set. Based on our results, we proposed an alternative model in which DBA with GATA1 mutations leads to diminished expression of GATA1 target genes, including many responsible for production of ribosomes, thus leading to insufficient translation and erythroid failure. In DBA patients with RP mutations, the primary deficiency of a ribosomal protein causes ribosomal deficiency and reduced translation leading to erythroid failure.²  The primary motivation for the reanalysis of our work by Ulirsch et al appears to be that we did not observe the strong GATA1-deficient signature in our RP patient samples. They proposed that excessive heterogeneity among the analyzed cells confounded the interpretation of our results.

To address heterogeneity, Ulirsch et al applied a complicated statistical deconvolution procedure using previously published data from 2 independent studies to classify cell mixtures that might have confounded our study (we note that none of the studies considered here uses the same marker sets although all can be reasonably expected to enrich a primarily erythroid subpopulation). Although they claim their method is robust, a careful analysis of their supplemental data⁴  suggests that this method has significant limitations for the purpose they propose. For example, using the classification algorithm of Ulirsch et al to compare the Merryweather-Clark et al⁵  based predictor to the erythroid expression data of Novershtern et al,⁶  each of the 3 more differentiated erythroid Novershtern populations (of 5 possible) had nearly identical abundance of all 4 of the possible Merryweather-Clark cell populations. Likewise, when the Novershtern predictor is applied to the Merryweather-Clark data, similar proportions of cell types are observed in both colony-forming unit erythroid and proerythroblast populations, including nearly 25% of cells with a mature phenotype. In fact, using either reference set to establish a classifier and then applying that classifier to the very same data set only predicts 60% to 85% identity for the cells within these carefully controlled populations. The simplest explanation for these discrepancies is that the classifier methods used by Ulirsch et al have limitations. Indeed, published reports have cast doubt on the utility of complex mixture deconvolutions, particularly in separating cell types with closely cocorrelated expression.⁷ 

When Ulirsch et al applied their classifying methods to the DBA progenitor transcriptional profiles in O’Brien et al,²  the results were inconsistent. Results from Ulirsch et al suggest 15% to 25% of cells in the CD44⁺CD235⁻ populations generated from the RP patients were late erythroid cells. Late erythroid cells defined in the Merryweather-Clark et al study had a mature morphologic phenotype, with marked nuclear pyknosis and clear evidence of cytoplasmic hemoglobinization. No such cells were observed in cytospin examinations of these cultures (supplemental Figure 1 in O’Brien et al² ). Similarly, Ulirsch et al suggested that cells expressing the short form of GATA1 (GATA1s) had an accumulation of early progenitors, in contrast to the morphological and surface phenotypes we reported.

Finally, to illustrate the potential problems associated with comparison of “imperfectly” stage-matched data, Ulirsch et al used their analyses to generate a synthetic normal set of comparators in which gene-expression signals were adjusted by the estimated expression of the putative fraction of admixed cells. We were gratified to see that, using the most favorable comparator for cell-composition adjustment (eg, the one with the greatest proportion of early progenitors in the GATA1 group and most differentiated progenitors in the RP group), the expression of GATA1 target genes remained upregulated. Ulirsch et al suggest that this discrepancy is “likely” ex vivo culture-based selection, although they do not provide an explanation as to why selection should be so different in progenitors with GATA1 and RP mutations when they are postulated to have directly linked pathophysiology. We feel that a more likely explanation, regardless of which model one finds credible, is that the mixture estimates derived by Ulirsch et al have limitations.

Our study used several layers of control in both process and analysis: surface marker–based sorting, morphologic examination, and stringent and rank-based gene-set evaluations. By these standards, the ex vivo data presented in Ludwig et al could be subject to the same confounding variables as our ex vivo data, including the stage of the knockdown and control cells, the drug selection of the transduced cells, and variable numbers of shRNA vectors and shRNA expression in the population of transduced cells. Importantly, their analysis of patient samples is complicated by analysis of only 3 DBA patients, two-thirds of whom were on treatment with steroids that had stimulated erythropoiesis at the time the sample was obtained.

The differences in interpretation seem to have provoked the passion in the Letter to Blood from Ulirsch et al. In our view, Ulirsch et al promote a 1-sided argument in which perfection has become the enemy of the good. Variation is inherent in biological experiments, and reasonable scientists must always debate about how to deal with experimental variability. It can be and should be questioned whether any cell from a DBA patient with a deficiency of mature 80S ribosomes, primary or cultured, erythroid or nonerythroid, can have a perfectly matched normal control. Philosophical debates about how to control for confounding variables, especially in the case of comparing a disease tissue to normal, are valuable to raise awareness within the field and to help improve the rigor of the analysis. We accept the commentary of Ulirsch et al in that spirit. But we believe that the best use of our time and resources would be through a collaboration to design, execute, and analyze experiments that will resolve the differences in the 2 models. We welcome the authors of Ulirsch et al and any others to join us in this endeavor.