Utility of caregiver signaling questions to detect neurocognitive impairment in children with sickle cell disease

Children with sickle cell disease (SCD) are at increased risk of disease-related neurocognitive dysfunction.^1,2 These cognitive difficulties directly impact academic performance, and children with SCD demonstrate heightened risk of learning difficulties and grade retention.^3,4 Thus, early identification of neurocognitive difficulties in children with SCD is essential to provide timely referrals to intervention services that may support more favorable outcomes.

The American Society of Hematology (ASH) cerebrovascular guidelines (Recommendations 8.1 and 8.3) recommend regular surveillance using simplified “signaling questions” to screen for possible neurocognitive and academic difficulties in patients with SCD.⁵ Referral for formal screening and neurocognitive testing is made upon abnormal surveillance results.⁵ This approach was adopted from the American Academy of Pediatrics, which recommends that developmental surveillance be incorporated into every well-child preventative care visit for the general population, with incorporation of formal, standardized screening measures if concerns arise during surveillance.⁶ This developmental surveillance and screening model has been adopted by several other initiatives for specific patient populations (eg, autism spectrum disorder, attention-deficit/hyperactivity disorder).^7-9 In general, neurocognitive surveillance and screening should be highly sensitive to reduce false negatives and thus the likelihood of failing to provide needed intervention services.^10,11 

The ASH guidelines provide sample signaling questions that pertain to academic, social, and behavioral development across the lifespan, which may be suggestive of possible neurocognitive deficits in patients with SCD. These questions were adapted from pre-existing protocols used with non-SCD populations.¹² Although these protocols have predictive validity and sensitivity to detect developmental delay within general and other non-SCD populations, some studies have noted that in isolation, developmental surveillance lacks sufficient sensitivity and specificity to reliably identify developmental delays.^13-16 Nonetheless, the clinical utility of these questions within SCD populations is unclear, and there are several considerations that may impact the functionality of signaling questions within a SCD population, compared with non-SCD populations. Most patients with SCD in the United States identify as Black or African American and are socioeconomically disadvantaged, with structural and interpersonal factors that may substantially affect their ability and likelihood to utilize health care services.^17-19 Similarly, accuracy of a caregiver’s report on their child’s developmental status may be limited by parental educational attainment, reading level, and perception of symptoms, which represent additional barriers to health care utilization in patients with SCD.²⁰ 

Only 2 studies have examined the use of signaling questions to identify neurocognitive impairment in children and adolescents with SCD.^21,22 One documented that children with SCD classified by their caregiver as being “below grade level” had poorer intellectual functioning, working memory, and processing speed relative to those described as “at or above grade level.”²¹ Caregiver ratings of their child being below grade level demonstrated good specificity for detecting cognitive impairment (>1 standard deviation [SD] below the normative mean) but low sensitivity.²¹ Another study showed that caregiver-reported academic or cognitive concern demonstrated a high number of false negatives and false positives for identifying academic or cognitive impairment in children with SCD. However, this study was limited by a small sample size, and signaling questions were administered via a written demographic form.²² Further research is needed to determine the most sensitive and specific caregiver signaling questions for identifying neurocognitive impairment in children with SCD, and to elucidate whether the use of signaling questions improves our ability to identify neurocognitive and academic impairment beyond known medical and demographic factors alone.

Our first objective was to examine the relationship between caregiver signaling questions and performance-based measures of neurocognitive functioning in children with SCD. Our second objective was to determine the most accurate signaling questions for identifying neurocognitive and academic impairment in children with SCD. Our third objective was to determine whether caregiver signaling questions would improve the accuracy of predicting neurocognitive and academic impairment, beyond clinical and demographic characteristics alone. We hypothesized that, after adjustment for demographic and medical factors, questions pertaining to a child’s academic development (ie, academic questions) would be predictive of neurocognitive impairment, emerge as the most accurate signaling questions, and improve the ability to identify neurocognitive and academic impairment beyond clinical/demographic factors alone. However, we did not expect the same predictive ability and accuracy with questions pertaining to social and behavioral development.

This cross-sectional cohort study was approved by the Institutional Review Board at St. Jude Children’s Research Hospital. Written informed consent was obtained in accordance with hospital policy. This study was conducted in accordance with the Declaration of Helsinki.

Children and adolescents with SCD enrolled in the Sickle Cell Clinical Research and Intervention Program (SCCRIP) were eligible to participate in this study. SCCRIP is a longitudinal cohort study that collects retrospective and prospective data on clinical, neurocognitive, geographical, psychosocial, and health outcomes of individuals with SCD.²³ Participants completed neurocognitive assessments as part of systematic surveillance and standard care, regardless of disease severity, cerebrovascular injury, or neurocognitive concern. Evaluations occurred approximately every 4 years, between the ages of 8 and 18 years, and included performance-based measures of cognitive and academic abilities, and caregiver ratings of emotion regulation, behavior, and adaptive function. For the present study, testing occurred at 1 of 3 developmental stages: school age (age 8-9 years), early adolescence (age 12-13 years), and late adolescence (age 16-17 years). Of 753 eligible SCCRIP participants, 421 completed neurocognitive evaluations (supplemental Table 1). The earliest (eg, youngest age) assessment after the age of 8 years was used in analyses for participants who completed multiple serial evaluations (n = 76), ensuring that no participants had a prior assessment completed through our program. This was done intentionally to avoid biasing caregiver response to signaling questions. Data were collected between June 2012 and December 2021. Testing completed during the COVID-19 pandemic was conducted in person with safety precautions consistent with institutional policy (ie, masks, social distancing).

During neurocognitive evaluations, primary caregivers were verbally asked a series of structured interview questions by a licensed psychologist or a trainee supervised by a licensed psychologist. These structured questions pertained to their child’s academic performance, social development, and behavior at school, and were agreed upon by licensed clinical psychologists. Caregivers were asked the following 7 questions: “What are your child’s grades in school?,” “Has your child ever had learning difficulties?,” “Do you have any concerns about your child’s learning?,” “Has your child ever repeated a grade?,” “Do you have any concerns about your child’s behavior at school?,” “Does your child generally get along well with peers?,” and “Do you have any concerns regarding your child’s social development?” Follow-up to structured interview questions was determined by the interviewing clinician according to clinical need.

Demographic, medical, and treatment variables were obtained from the SCCRIP database. The Social Vulnerability Index (SVI) was calculated for each participant and used to measure social vulnerability and household socioeconomic status.²⁴ The SVI classifies individuals on the basis of socioeconomic disadvantage at the neighborhood level (eg, poverty, education, and housing data), with higher scores indicating greater vulnerability. Participants with hemoglobin SS (HbSS) or hemoglobin S-beta zero (HbSβ⁰) thalassemia were treated with hydroxyurea according to National Heart, Lung, and Blood Institute guidelines.²⁵ For participants with hemoglobin S-C (HbSC) or hemoglobin S-beta plus (HbSβ⁺) thalassemia, hydroxyurea initiation was guided by the frequency of acute disease complications.²⁶ Laboratory values were collected on the day of neurocognitive evaluation or, if unavailable, the closest blood draw before testing.

The Wechsler Abbreviated Scale of Intelligence, Second Edition was used to assess verbal comprehension (Verbal Comprehension Index) and perceptual reasoning (Perceptual Reasoning Index).²⁷ Working memory and graphomotor speed were measured using the Digit Span and Coding subtests, respectively, from the Wechsler Adult Intelligence Scale, Fourth Edition or Wechsler Intelligence Scale for Children, Fourth or Fifth Editions, depending on participant age.^28,29 Word reading and math fluency skills were assessed using the Woodcock-Johnson Tests of Achievement, Third or Fourth Editions, Letter-Word Identification and Math Fluency subtests).²⁴ Administration of all neurocognitive measures was supervised by a licensed psychologist.

Signaling questions were analyzed as categorical variables. Caregiver responses to current or past learning difficulties, history of grade retention, ability to get along with peers, and concerns about their child’s learning, behavior at school, and social development were categorized as “yes” or “no.” Caregiver report on their child’s academic grades were categorized as "primarily As or Bs" or "primarily Cs, Ds, Fs, or variable". Standard scores or scaled scores from neurocognitive measures falling >1.5 and >1 SD below the normative mean were used as cutoffs for clinical impairment.^30-32 Participants were classified as having “significant” or “mild” neurocognitive impairment if they performed >1.5 or >1 SD below the normative data on ≥2 out of 6 neurocognitive measures, respectively.^33,34 With the present 6 neurocognitive measures, the estimated likelihood in the general population of being classified as having significant or mild neurocognitive impairment is approximately 5.6% or 24.4%, respectively, using this criterion. Each test-specific neurocognitive measure >1.5 or >1 SD below the normative mean was also analyzed as a secondary outcome. Summary statistics were compared across 3 age groups using the χ² test for categorical variables and Kruskal-Wallis rank sum test for continuous variables, respectively.

For the first objective, we used multivariable generalized linear regression models with a binomial link function to evaluate the relationships between caregiver responses to signaling questions and neurocognitive impairment (≥2 out of 6 neurocognitive measures with >1.5 or >1 SD below the normative mean). Age, SVI, genotype, and hydroxyurea treatment were examined as covariates in primary data analysis given associations with disease progression and cognitive outcomes.¹ Sex was also included as a covariate in models to account for differences in cognitive outcomes in the general population.^35,36 All the covariates were tested for multicollinearity before entering the model (a variance inflation factor <2). For our second objective, we calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of each signaling question for detecting neurocognitive impairment (≥2 out of 6 neurocognitive measures with >1.5 or >1 SD below the normative mean). Signaling questions exhibiting significant associations with neurocognitive impairment were incorporated into the final multivariate model. Concern for learning was removed because it was very strongly correlated with history of learning difficulties (variance inflation factor >2).

We generated the areas under the receiver operating characteristic (ROC) curves (AUCs) to estimate whether adding these significant caregiver signaling questions into the model could improve the accuracy of predicting neurocognitive impairment beyond the use of clinical characteristics alone. We calculated 95% confidence intervals and compared 2 ROC curves using the DeLong method.³⁷ We developed a cumulative risk score (CRS) aimed at predicting neurocognitive impairment, based on the identified association signals to enhance the clinical applicability of the signaling questions. The CRS aggregated the number of risk factors, including a history of grade retention, history of learning difficulties, and poor grades (Cs, Ds, Fs, variable), and was analyzed using an additive dose-effect model. The CRS ranged from 0 to 3, with a higher score indicating a higher number of positive results, increased caregiver concern, and an increased risk of impairment. We also generated ROCs for the CRS-only model and compared it with the combined model with both CRS and clinical characteristics to evaluate its relative performance. Given differences in developmental expectations based on age, follow-up analyses were separately conducted by age (school age, early adolescence, late adolescence) across both objectives. Complete-case analysis without imputation was performed.

For the first objective, false discovery rate–adjusted P values or q values were calculated across all signaling questions and all neurocognitive outcomes to correct for multiple comparisons. Q value <0.05 was considered significant in our primary analyses. Otherwise, a P value <.05 was considered statistically significant. Scores reflecting significant neurocognitive impairment (>1.5 SD below the normative mean on ≥2 neurocognitive measures) were our primary outcome of interest, consistent with traditional definitions of impairment, although results reflecting mild neurocognitive impairment (>1 SD below the normative mean on ≥2 neurocognitive measures) are also briefly reported.^33,34 

As displayed in Table 1, a total of 421 participants received a neurocognitive evaluation, and the mean age at time of testing was 12.0 years (SD, 3.4 years). Approximately 46% (n = 192), 16% (n = 69), and 38% (n = 160) of participants fell into the school age, early adolescent, and late adolescent groups, respectively. All but 3 (99%) participants identified as Black. The HbSS/Sβ⁰ thalassemia genotype was most prevalent (62%). Those that received neurocognitive testing were statistically comparable to those who did not, except the latter were significantly younger and displayed greater social vulnerability (supplemental Table 1). Out of 421 participants, 260 (62%) participants had data on all 6 neurocognitive measurements that were used to define significant and mild neurocognitive impairment. These participants were statistically comparable to those who did not have data on all 6 neurocognitive measurements, except the latter were on average 1 year older (supplemental Table 2).

As displayed in Table 2, ∼40% of caregivers reported that their child had a history of learning difficulties (n = 157), and 36% reported concern for their child’s learning (n = 132). Most caregivers reported that their child earns primarily As or Bs (58%; n = 201), and 21% were reported to have repeated a grade (n = 79).

Approximately 10% to 20% of patients with SCD obtained a score >1.5 SD below the normative mean across tasks of verbal comprehension, perceptual reasoning, working memory, graphomotor speed, and word reading (Table 2). Approximately 33% obtained a score below this cutoff on a task of math fluency (Table 2). Descriptive statistics for outcomes using a more modest cutoff (>1 SD below the normative mean) can be found in Table 2. Overall, 74 (28%) and 141 (54%) participants were classified as having significant and mild neurocognitive impairment, respectively. The number of patients with SCD categorized into the significant or mild neurocognitive impairment group increased with age (Table 2).

Associations between caregiver-reported signaling questions and neurocognitive and academic impairment

Associations between signaling questions and performance measures after controlling for confounding variables are displayed in Table 3. Children who were reported to have repeated a grade, did not obtain primarily A and B grades in school, had a history of learning difficulties, or whose caregiver reported concern for their learning were more likely to be classified as having significant neurocognitive impairment and obtained lower scores (>1.5 SD below the normative mean) across all neurocognitive and academic measures (all q < 0.05). Caregiver concern for a child’s social development was associated with math fluency performance (q = 0.049); however, caregiver concerns for their child’s behavior at school, social development, or ability to get along with peers were not associated with neurocognitive or academic outcomes (all other q > 0.05). Results reflecting mild neurocognitive impairment (>1 SD below the normative mean) were largely consistent with the above results (Table 3).

Sensitivity, specificity, PPV, and NPV of signaling questions for detecting neurocognitive and academic impairment in the total sample

Sensitivity and specificity analyses were conducted for signaling questions, with neurocognitive impairment defined as >1.5 SD below the normative mean (Table 4). For these analyses, caregiver report of history of learning difficulties (sensitivity, 0.64; specificity, 0.77) emerged as the most sensitive and specific signaling question for detecting significant neurocognitive impairment and domain-specific impairment (sensitivity range, 0.56-0.77; specificity range, 0.63-0.72). Across all neurocognitive domains, caregiver report of history of learning difficulties (sensitivity, 0.77; specificity, 0.67), academic grades (sensitivity, 0.74; specificity, 0.64), and concerns for learning (sensitivity, 0.75; specificity, 0.71) were most sensitive and specific for detecting word reading impairment. PPVs and NPVs of academic/learning signaling questions for significant neurocognitive impairment ranged from 0.42 to 0.61 and 0.81 to 0.86, respectively (supplemental Table 3).

Analyses were repeated with neurocognitive or academic impairment defined as >1 SD below the normative mean. Results were broadly consistent with initial calculations when using the mild neurocognitive impairment cutoff (Table 4). PPVs and NPVs of academic/learning signaling questions for mild neurocognitive impairment ranged from 0.64 to 0.91 and 0.55 to 0.62, respectively (supplemental Table 3).

Sensitivity and specificity of signaling questions for detecting neurocognitive and academic impairment stratified by age

Sensitivity, specificity, PPV, and NPV analyses were also conducted within each age group (school age, early adolescence, late adolescence) (supplemental Tables 4-9). Of note, analyses using the >1.5 SD below the normative mean cutoff were limited by a small sample size within some age groups. Across age groups, significant neurocognitive impairment was most consistently detected by caregiver academic signaling questions surrounding history of learning difficulties (sensitivity range, 0.52-0.86; specificity range, 0.67-0.81), concerns for learning (sensitivity range, 0.46-0.86; specificity range, 0.72-0.83), and academic grades (sensitivity range, 0.60-0.75; specificity range, 0.70-0.75). PPVs and NPVs of academic/learning signaling questions for significant neurocognitive impairment ranged from 0.30 to 0.75 and 0.64 to 0.96 across age groups, respectively. Analyses using the >1 SD below the normative mean cutoff were broadly consistent with initial calculations (supplemental Tables 4-9).

To examine the potential utility of combining signaling questions, we built the CRS (see “Methods”) using history of grade retention, history of learning difficulties, or academic grades (As or Bs vs others), and compared its predictive accuracy with the final full model including both CRS and demographic/clinical factors (Figures 1-3; supplemental Tables 10 and 11). Figure 1 depicts the area under the curve for the relationship between demographic/clinical characteristics, CRS, and significant/mild neurocognitive impairment (>1.5 or >1 SD below the normative mean on ≥2 out of 6 measures). These AUC curves demonstrate that a child’s caregiver-reported cumulative risk (CRS) improved the prediction of significant or mild neurocognitive impairment, beyond known demographic/clinical characteristics alone. However, the final full model, which included both demographic/clinical characteristics and CRS, did not improve prediction/classification accuracy compared with the CRS-only model (significant impairment AUC, 0.81; mild impairment AUC, 0.76).

We also examined the clinical applicability of these models at the domain-specific level for each individual measure. As shown in Figure 2, using the 1.5 SD below the normative mean cutoff, cumulative risk (CRS) significantly improved sensitivity and specificity for detecting all domains (maximum P = .037 and minimum P = .00017) compared with demographic/clinical characteristics alone. Combined CRS and clinical models improved prediction accuracy compared with CRS-only models for detecting word reading (P = .01) and math fluency (P = .014) impairment, but no other domains (P ≥ .09). AUC curves showing the relationship between demographic/clinical characteristics, CRS, and domain-specific impairment using the >1 SD below the normative mean cutoff were consistent with initial analyses and are depicted in Figure 3.

The ASH cerebrovascular disease guidelines for SCD recommend regular surveillance using simplified signaling questions to screen for neurocognitive difficulties to aid in early detection (Recommendations 8.1 and 8.3), although the clinical utility of these signaling questions has yet to be established.⁵ In the present study, caregiver signaling questions pertaining to their child’s school performance were separately associated with significant and mild neurocognitive impairment, and domain-specific impairment on individual measures. Most individual signaling questions displayed lower sensitivity (eg, <0.6) for detecting neurocognitive impairment than is expected for performance-based neurocognitive screening tools, which typically demonstrate sensitivity >0.85.^10,34,38 However, cumulative caregiver report (ie, sum of reported signaling questions) improved prediction of neurocognitive and academic impairment beyond known demographic/clinical factors alone, demonstrating the value of signaling questions in aiding the detection of patients needing support.

These findings are consistent with prior studies that have evaluated the use of caregiver academic signaling questions used in isolation.^21,22 The sensitivity of these questions may be related to several factors, including higher social vulnerability, stigma, medical distrust, parental educational attainment, or bias in the perception of symptoms.²⁰ Notably, academic signaling questions were more sensitive and specific for detecting academic concerns, rather than cognitive concerns, particularly for word reading. Sensitivity improved when patients were stratified into age groups, with higher sensitivity in older patients with SCD, which likely reflects greater caregiver exposure to their child’s academic difficulties due to longer enrollment in school and cumulative disease progression. Sensitivity and specificity were broadly consistent for both mild and significant neurocognitive impairment. Caregiver signaling questions were more accurate in ruling out significant neurocognitive impairment (NPV range, 0.81-0.86) compared with mild neurocognitive impairment (NPV range, 0.55-62) because of the lower frequency of significant impairment. It has yet to be established if these results are unique to the SCD population or would be observed in demographically similar non-SCD populations, although there is research to suggest that the predictive validity of parent report may be influenced by demographic and environmental factors.³⁹ 

Medical providers most often rely conjointly on caregiver report and medical/demographic information to make appropriate recommendations. Consistent with predictions, cumulative caregiver report improved classification accuracy for academic domains and significant neurocognitive impairment, over and above that of known demographic or medical factors. This suggests that the addition of caregiver signaling questions to routine medical practice for patients with SCD may improve the ability to identify those in need of additional support.

Patients with SCD encounter a host of barriers that make it difficult to access and utilize health care services (eg, structural racism, parental education, literacy, perception of symptoms).^17-19 Across most settings, individuals with SCD are only referred for formal screening when caregivers or providers note concerns. Incorporation of caregiver signaling questions into patient care offers the opportunity to improve access and utilization of health care services for patients with SCD. Although direct, brief performance-based screening for children and adolescents with SCD is the most sensitive means to ensure that all patients with SCD with neurocognitive or academic impairments receive proper intervention, universal screening for all children and adolescents with SCD requires ample resources (eg, time, expertise) that are often unavailable. For resource-rich institutions with access to such services, implementation of a universal screening model will be the most effective means to identify patients with SCD in need of support. Several institutions have already implemented universal SCD screening programs for children and adolescents with SCD.^22,40 However, for such institutions that do not have access to needed resources, use of caregiver signaling questions is a brief and low-cost method that can be easily and quickly implemented into patient care. Likewise, our results suggest that incorporation of these signaling questions is comparably better than demographic/clinical information alone for accurately identifying patients with SCD with cognitive or academic impairment. Incorporation of signaling questions into routine medical checkups will allow for more regular, longitudinal surveillance given repeated practice across clinical encounters. Although the current study was only able to include a single comparison point for caregiver signaling questions, sensitivity in accurately identifying patients with true impairment may increase with serial surveillance. Future research should evaluate the utility of caregiver signaling questions across several encounters for detecting neurocognitive impairment longitudinally.

This study has several strengths. Performance-based neurocognitive and academic outcomes and caregiver responses to academic, social, and behavioral signaling questions were available for a large cohort of children and adolescents with SCD. The concurrent collection of these 2 sources of information allowed us to examine the utility of signaling questions relative to performance-based measures across several developmental time points. However, several limitations should be considered. A comprehensive assessment of each child’s reading and math abilities was not completed, and it is possible that children struggling with more advanced reading or math skills (eg, reading comprehension, applied math problems) were not identified as experiencing academic difficulties by direct performance-based testing. Additionally, formal school performance data were not available for each participant beyond caregiver report. A demographically matched or sibling control group was not included. Furthermore, families who canceled or missed numerous appointments and consequently did not participate may have contributed to a biased and nonrepresentative sample, given that those who did not participate displayed greater social vulnerability. Neuroimaging was not conducted for all patients, and therefore, the potential impact of silent cerebral infarcts is unknown. Several of our analyses were also restricted by a small sample size. Finally, the included signaling questions were primarily focused on learning and academic outcomes. Future research should investigate the usefulness of broader cognitive signaling questions for detecting neurocognitive or academic impairment in patients with SCD.

Early detection of disease-related neurocognitive and academic impairment is essential to provide timely intervention for patients with SCD. Caregiver signaling questions improved accurate identification of patients with SCD and neurocognitive impairment but displayed lower sensitivity than would be expected for performance-based neurocognitive screening tools. Signaling questions represent a brief and low-cost method that can be implemented universally to improve the detection of neurocognitive impairment in patients with SCD.

The authors thank the patients and caregivers who participated in this study. The authors also acknowledge the valuable contributions of Jason Hodges, Pei-Lin Chen, Courtney Mays, Erin MacArthur, Madelene Wilson, Tiana Thomas, Ruth Johnson, and Michelle Brignac, who all are affiliated with and employed by St. Jude Children's Research Hospital, in preparing and analyzing the data.

This work was supported by the American Lebanese Syrian Associated Charities. A.A.K. (K12HL137942 and K24HL148305) and A.M.H. (K23HL166697) were supported by the National Heart, Lung, and Blood Institute during this study. The remaining authors received no additional funding.

Contribution: C.M.D. interpreted the data and drafted and revised the manuscript; L.K. interpreted the data and drafted the initial manuscript; J.G. analyzed the data, assisted with data interpretation, and critically reviewed the manuscript; J.N.L, B.P., A.A.K., C.M.T., and J.S.H. contributed to study design and critically reviewed the manuscript; G.K. assisted with study design, analyzed the data, assisted with data interpretation and drafting of the manuscript, and critically reviewed the manuscript; A.M.H. contributed to study design, interpreted the data, and critically reviewed the manuscript; and all authors approved the final manuscript as submitted and agreed to be accountable for all aspects of the work.

Conflict-of-interest disclosure: The authors declare no competing financial interests.

Correspondence: Andrew M. Heitzer, St. Jude Children’s Research Hospital, 262 Danny Thomas Place, MS 740, Memphis, TN 38105-3678; email: aheitzer@stjude.org.