Fully automated volumetric measurement of malignant pleural mesothelioma by deep learning AI: validation and comparison with modified RECIST response criteria

Our plain-English summary of Fully automated volumetric measurement of malignant pleural mesothelioma by deep learning AI: validation and comparison with modified RECIST response criteria, by A.C. Kidd, O. Anderson, G.W. Cowell, et al (Thorax, 2022).

The study

• Authors: A.C. Kidd, O. Anderson, G.W. Cowell, A.J. Weir, J.P. Voisey, M. Evison, S. Tsim, K.A. Goatman, K.G. Blyth
• Journal: Thorax
• Published: 2022;77(12):1251-1259
• Cited by: 36 articles, as counted by NIH iCite
• Read it: PubMed · Publisher · Free full text

Abstract

Background: In malignant pleural mesothelioma (MPM), complex tumour morphology results in inconsistent radiological response assessment. Promising volumetric methods require automation to be practical. We developed a fully automated Convolutional Neural Network (CNN) for this purpose, performed blinded validation and compared CNN and human response classification and survival prediction in patients treated with chemotherapy.

Methods: In a multicentre retrospective cohort study; 183 CT datasets were split into training and internal validation (123 datasets (80 fully annotated); 108 patients; 1 centre) and external validation (60 datasets (all fully annotated); 30 patients; 3 centres). Detailed manual annotations were used to train the CNN, which used two-dimensional U-Net architecture. CNN performance was evaluated using correlation, Bland-Altman and Dice agreement. Volumetric response/progression were defined as ≤30%/≥20% change and compared with modified Response Evaluation Criteria In Solid Tumours (mRECIST) by Cohen's kappa. Survival was assessed using Kaplan-Meier methodology.

Results: Human and artificial intelligence (AI) volumes were strongly correlated (validation set r=0.851, p<0.0001). Agreement was strong (validation set mean bias +31 cm3 (p=0.182), 95% limits 345 to +407 cm3). Infrequent AI segmentation errors (4/60 validation cases) were associated with fissural tumour, contralateral pleural thickening and adjacent atelectasis. Human and AI volumetric responses agreed in 20/30 (67%) validation cases κ=0.439 (0.178 to 0.700). AI and mRECIST agreed in 16/30 (55%) validation cases κ=0.284 (0.026 to 0.543). Higher baseline tumour volume was associated with shorter survival.

Conclusion: We have developed and validated the first fully automated CNN for volumetric MPM segmentation. CNN performance may be further improved by enriching future training sets with morphologically challenging features. Volumetric response thresholds require further calibration in future studies.

© Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

In plain English

This multicentre retrospective cohort study tested a deep learning artificial intelligence (AI) program that measures the volume of malignant pleural mesothelioma tumour on CT scans with no human input. The study did not examine asbestos exposure. The AI was trained on 123 CT datasets from 108 patients in Glasgow. It was then tested on 60 scans from 30 patients at three UK centres, taken before and after cisplatin or carboplatin with pemetrexed chemotherapy. In the test set, AI and human volumes were strongly correlated (r=0.851), and their average difference of +31 cm3 was not statistically significant. Large errors occurred in 4 of the 60 test scans. AI and human volume readings put patients in the same response category in 20/30 cases (67%). AI and the standard mRECIST method agreed in 16/30 cases (printed as 55%), which the authors called fair agreement. Higher tumour volume before chemotherapy strongly predicted shorter survival. The authors concluded they had developed and validated the first fully automated tool of this kind. They said the volume thresholds used to classify response need further calibration and that their sample size was modest for deep learning.

We wrote this plain-English summary from the full article with the help of AI, and a second AI model checked each sentence against the article. It is not the authors' wording; the abstract and the article are their own account.

What the study did and found

• Design: Multicentre retrospective cohort study developing and validating a fully automated deep learning (convolutional neural network) tool for measuring mesothelioma tumour volume on CT
• Who was studied: Patients with malignant pleural mesothelioma (MPM) in the UK. The training and internal validation set had 123 CT datasets from 108 patients in Glasgow (80 fully annotated). The external validation set had 60 CT datasets from 30 patients from Leicester (10), Manchester (10) and Glasgow (10), each with a scan before and after chemotherapy. Study dates are not given.
• Exposure: Asbestos exposure was not studied. The study measured tumour volume on CT scans taken before and after cisplatin or carboplatin with pemetrexed chemotherapy.
• Main results:
  • Training set: human and AI tumour volumes were strongly correlated (r=0.847, p less than 0.0001).
  • Training set: AI volumes were significantly larger than human volumes, with a mean bias of +142 cm3 (p less than 0.0001, 95% limits −226 to 511 cm3).
  • Agreement between two human readers was moderate (ICC 0.732, p=0.001). Region overlap between the reference human annotation and the AI (mean Dice 0.54) was better than with the second human reader (mean Dice 0.36, p=0.002). It was worse than the reference reader's own re-annotation (mean Dice 0.61, p=0.014).
  • External validation set (60 scans, 30 patients): human and AI volumes were strongly correlated (r=0.851, p less than 0.0001). The mean bias was +31 cm3, not significantly different from zero (p=0.182), with 95% limits of −345 to +407 cm3.
  • In 4/60 validation scans the AI-human difference fell outside the 95% limits. In two scans the AI missed fissural tumour. In the other two it wrongly included collapsed (atelectatic) lung over the diaphragm, or benign pleural thickening on the opposite side.
  • Tumour volumes trended lower after chemotherapy, but neither change was statistically significant: human 366 vs 328 cm3, p=0.196; AI 427 vs 371 cm3, p=0.081.
  • Human and AI volumetric response categories (partial response, stable disease, progressive disease) agreed in 20/30 (67%) validation cases, kappa=0.439 (0.178 to 0.700). Simplified to progression versus no progression, they agreed in 26/30 (87%), kappa=0.586 (0.227 to 0.945).
  • mRECIST and AI response categories agreed in 16/30 (printed as 55%) validation cases, kappa=0.284 (0.026 to 0.543), described as fair agreement. Simplified to progression versus no progression, they agreed in 20/30 (67%), kappa=0.223 (−0.128 to 0.574).
  • In the validation cohort (median overall survival 377 days), higher tumour volume before chemotherapy, by human or AI measurement, was a strong predictor of overall survival.
  • Progressive disease versus no progression was linked to non-significantly shorter survival. By mRECIST this was 293 vs 399 days (HR 1.78 (0.71 to 4.46), p=0.149, n=26). By human volumes it was HR 1.61 (0.51 to 5.07), p=0.326, and by AI volumes HR 1.58 (0.37 to 6.75), p=0.326.
• The authors' conclusion: The authors concluded they had developed and validated the first fully automated deep learning CNN for volumetric assessment of malignant pleural mesothelioma.
• Limitations the authors state:
  • The sample size was modest compared with many deep learning projects.
  • The authors said the poor agreement between AI and mRECIST response most likely reflects poor calibration of the volumetric response thresholds chosen.
  • The tool is limited by the clarity of the CT images it is given.
  • The authors suggested the positive bias of AI volumes in the training set may reflect a mix of disease stages in that set.
  • Four of the 30 validation cases were excluded from the response-survival analyses because the post-chemotherapy scan came more than 100 days after treatment. The authors called this threshold set in advance but arbitrary.
  • The study was exploratory, and no sample size calculation was done.

Topics

The National Library of Medicine indexes this study under subjects our site covers: Mesothelioma.

Medical terms in this study, explained in our Lexicon: Pleural Mesothelioma, Modified RECIST (mRECIST), Chemotherapy, Cohort Study, Pleural Thickening, Atelectasis.

Related studies

• Deep Learning for the Automatic Quantification of Pleural Plaques in Asbestos-Exposed Subjects — I. Benlala, B.D. De Senneville, et al. (2022)
• Deep-learning based classification distinguishes sarcomatoid malignant mesotheliomas from benign spindle cell mesothelial proliferations — J.R. Naso, A.B. Levine, et al. (2021)
• Asbestos Detection with Fluorescence Microscopy Images and Deep Learning — C. Cai, T. Nishimura, et al. (2021)
• Comparison of modified Response Evaluation Criteria in Solid Tumors, European Organization for Research and Treatment of Cancer criteria, and PET Response Criteria in Solid Tumors for evaluation of tumor response to chemotherapy and prognosis prediction in patients with unresectable malignant pleural mesothelioma — K. Kitajima, M. Maruyama, et al. (2020)
• Revised Modified RECIST Criteria in Malignant Pleural Mesothelioma (Version 1.1): A Step Forward in a Long Race — A.S. Tsao, G.W. Gladish, et al. (2018)
• Metabolic response assessment with 18F-FDG-PET/CT is superior to modified RECIST for the evaluation of response to platinum-based doublet chemotherapy in malignant pleural mesothelioma — S. Kanemura, K. Kuribayashi, et al. (2017)

Browse more research in Thorax, or return to medical research.