A Comparative Analysis of Individual Readers and an Autonomous Deep-Learning-Based Algorithm for Assessing Total Metabolic Tumor Volume from Positron Emission Tomography/Computed Tomography Scans

Introduction

Artificial intelligence-based tools are increasingly used to assist radiologists with assessing a patient's disease status and evaluating treatment effectiveness. However, before these tools can be applied in clinical trials or routine practice, robust measures must be established to evaluate their performance. Total metabolic tumor volume (TMTV) is a volumetric tumor burden assessment derived from positron emission tomography/computed tomography (PET/CT) scans that has demonstrated prognostic value in patients with B-cell non-Hodgkin lymphoma. In this study, we investigate the variability among human readers in quantifying TMTV from PET/CT scans of patients with lymphoma, then compare their performance with a fully automated deep-learning-based algorithm (Jemaa et al. J Digit Imaging 2020).

Methods

Diagnostic quality PET/CT scans at screening or at first on-treatment tumor response assessment (FOT) visits were obtained from 125 patients enrolled in an ongoing phase 1/2 multicenter, open-label, dose escalation and expansion study in patients with relapsed or refractory B-cell non-Hodgkin lymphoma (ClinicalTrials.gov identifier, NCT02500407). One scan per patient was selected at random from either time point. Manual TMTV (mTMTV) was assessed individually by 3 radiologists using a semiautomated reference region-based threshold method (1.5 × mean liver standardized uptake value + 2 standard deviations [SDs]). A consolidated ground truth read was created for each scan from the 3 mTMTV reads using a majority-vote approach. Automated TMTV (aTMTV) was applied to the same images, and aTMTV and mTMTV values were compared.

All TMTV values were cubic root-transformed for statistical analysis. Bland-Altman analysis and Dice similarity coefficients (DSCs) were used to assess concordance among mTMTV reads and between aTMTV and mTMTV reads, including ground truth. Correlation between aTMTV and the ground truth mTMTV was assessed using Pearson's correlation coefficient (r). Subgroup analyses compared aTMTV reads with ground truth mTMTV reads in patients of different age, sex, cancer subtype and prior therapy lines at baseline.

Results

In total, 110 images comprising 78 screening and 32 FOT scans from patients with diverse lymphoma types met image quality requirements and were included (aggressive lymphomas, n = 67; indolent lymphomas, n = 43). Bland-Altman analyses revealed good agreement between individual mTMTV reads of TMTV: the mean differences between mTMTV readers 1 versus 2, 2 versus 3, and 3 versus 1 were −0.02 (limit of agreement [LoA]: −1.89, 1.84), −0.03 (LoA: −1.65, 1.59) and 0.05 (LoA: −2.14, 2.24), respectively. A high degree of overlap was also observed between lesion segmentations, with DSCs of 0.89 (SD: 0.17) for reader 1 versus 2, 0.90 (SD: 0.16) for reader 2 versus 3, and 0.89 (SD: 0.19) for reader 3 versus 1.

Slightly greater measurement variability was observed when individual mTMTV reads were compared with aTMTV. The mean differences between methods were 0.10 (LoA: −1.53, 1.73) for reader 1 versus aTMTV, 0.12 (LoA: −1.75, 1.99) for reader 2 versus aTMTV, and 0.15 (LoA: −1.97, 2.28) for reader 3 versus aTMTV. DSCs for readers 1, 2 and 3 versus aTMTV were 0.70 (SD: 0.21), 0.70 (SD: 0.22) and 0.71 (SD: 0.22), respectively.

Good agreement and a strong correlation were observed between the consolidated ground truth mTMTV read and aTMTV (mean difference between methods: 0.05 [95% confidence interval (CI): −0.12, 0.22; LoA: −1.71, 1.80]; r = 0.97 [95% CI: 0.97, 0.98]). However, slight differences were observed in the spatial overlap of lesion segmentations between aTMTV and ground truth mTMTV (DSC: 0.71; SD: 0.22). Consistent results were observed across all patient subgroups.

Conclusion

The findings of this study provide valuable insights into the comparative performance of manual readers and aTMTV when assessing TMTV from PET/CT scans. aTMTV demonstrated similar performance to expert radiologists using semiautomated software for TMTV estimation, although slightly different regions were identified as lesions by the algorithm. Further research is warranted to explore the clinical implications of these findings. Understanding the degree of measurement variability among manual readers and how algorithm-based approaches differ may ultimately contribute to improved accuracy of automated TMTV assessments and support clinical adoption.

Xu:F. Hoffmann-La Roche AG: Current Employment, Current holder of stock options in a privately-held company. Jemaa:F. Hoffmann-La Roche AG: Current holder of stock options in a privately-held company; Genentech, Inc: Current Employment, Patents & Royalties: Patents pending. Kumar:Genentech, Inc: Current Employment. Balasubramanian:Genentech, Inc: Current Employment, Current holder of stock options in a privately-held company; F. Hoffmann-La Roche AG: Current holder of stock options in a privately-held company. Ounadjela:F. Hoffmann-La Roche AG: Current holder of stock options in a privately-held company; Genentech, Inc: Current Employment. Malik:Genentech, Inc: Current Employment; F. Hoffmann-La Roche AG: Current holder of stock options in a privately-held company. Lee:F. Hoffmann-La Roche AG: Current holder of stock options in a privately-held company; Genentech, Inc: Current Employment. Davis:Genentech, Inc: Current Employment. Pathania:F. Hoffmann-La Roche AG: Consultancy; Genentech, Inc: Consultancy. Wei:Genentech, Inc.: Current Employment, Patents & Royalties; F. Hoffmann-La Roche Ltd: Current equity holder in publicly-traded company. Ma:Genentech, Inc.: Current Employment; F. Hoffmann-La Roche Ltd: Current equity holder in publicly-traded company. Cybulski:Genentech, Inc.: Current Employment; F. Hoffmann-La Roche Ltd: Current equity holder in publicly-traded company. Carano:Genentech, Inc: Current Employment; F. Hoffmann-La Roche AG: Current holder of stock options in a privately-held company, Patents & Royalties: Patents issued/pending. Kostakoglu Shields:F. Hoffmann-La Roche AG: Consultancy; Genentech, Inc: Consultancy. Capra:Genentech, Inc: Current Employment; F. Hoffmann-La Roche AG: Current holder of stock options in a privately-held company.