Prospective validation of an artificial intelligence assessment in a cohort of applicants seeking financial compensation for asbestosis (PROSBEST)

Our plain-English summary of Prospective validation of an artificial intelligence assessment in a cohort of applicants seeking financial compensation for asbestosis (PROSBEST), by I. Smesseim, K.B.W.G. Lipman, S. Trebeschi, et al (European Radiology Experimental, 2025).

The study

• Authors: I. Smesseim, K.B.W.G. Lipman, S. Trebeschi, M.M. Stuiver, R. Tissier, J.A. Burgers, C.J. de Gooijer
• Journal: European Radiology Experimental
• Published: 2025;9(1):76
• Cited by: 2 articles, as counted by NIH iCite
• Read it: PubMed · Publisher · Free full text

Abstract

Background: Asbestosis, a rare pneumoconiosis marked by diffuse pulmonary fibrosis, arises from prolonged asbestos exposure. Its diagnosis, guided by the Helsinki criteria, relies on exposure history, clinical findings, radiology, and lung function. However, interobserver variability complicates diagnoses and financial compensation. This study prospectively validated the sensitivity of an AI-driven assessment for asbestosis compensation in the Netherlands. Secondary objectives included evaluating specificity, accuracy, predictive values, area under the curve of the receiver operating characteristic (ROC-AUC), area under the precision-recall curve (PR-AUC), and interobserver variability.

Materials And Methods: Between September 2020 and July 2022, 92 adult compensation applicants were assessed using both AI models and pulmonologists' reviews based on Dutch Health Council criteria. The AI model assigned an asbestosis probability score: negative (< 35), uncertain (35-66), or positive (≥ 66). Uncertain cases underwent additional reviews for a final determination.

Results: The AI assessment demonstrated sensitivity of 0.86 (95% confidence interval: 0.77-0.95), specificity of 0.85 (0.76-0.97), accuracy of 0.87 (0.79-0.93), ROC-AUC of 0.92 (0.84-0.97), and PR-AUC of 0.95 (0.89-0.99). Despite strong metrics, the sensitivity target of 98% was unmet. Pulmonologist reviews showed moderate to substantial interobserver variability.

Conclusion: The AI-driven approach demonstrated robust accuracy but insufficient sensitivity for validation. Addressing interobserver variability and incorporating objective fibrosis measurements could enhance future reliability in clinical and compensation settings.

Relevance Statement: The AI-driven assessment for financial compensation of asbestosis showed adequate accuracy but did not meet the required sensitivity for validation.

Key Points: We prospectively assessed the sensitivity of an AI-driven assessment procedure for financial compensation of asbestosis. The AI-driven asbestosis probability score underperformed across all metrics compared to internal testing. The AI-driven assessment procedure achieved a sensitivity of 0.86 (95% confidence interval: 0.77-0.95). It did not meet the predefined sensitivity target.

© 2025. The Author(s).

In plain English

This prospective study tested an artificial intelligence (AI) tool meant to support asbestosis compensation decisions in the Netherlands. It included 92 adults who applied for asbestosis compensation to the Institute for Asbestos Victims. Eighteen screened applicants were excluded for insufficient asbestos exposure. The article does not name fiber types or jobs. The AI tool combined a computer reading of chest CT scans with a lung function measure (DLCO). Each case was also judged by three pulmonologists, whose verdict served as the reference. For 62 applicants with an uncertain AI score, two more pulmonologists reviewed the case. The full AI-driven procedure had a sensitivity of 0.86 (95% CI 0.77 to 0.95), short of the preset 98% target, and a specificity of 0.85 (0.76 to 0.97). Agreement among the three reference pulmonologists (Randolph kappa) was 0.59 (95% CI 0.46 to 0.71) in 91 applicants, as one reviewer's verdict was missing. The authors concluded that the AI test did not meet the predefined criteria for successful validation. They noted that the study covered only Dutch applicants and that the tool's developers were partly involved.

We wrote this plain-English summary from the full article with the help of AI, and a second AI model checked each sentence against the article. It is not the authors' wording; the abstract and the article are their own account.

What the study did and found

• Design: prospective observational diagnostic accuracy study comparing an AI-driven index test with a reference panel of three pulmonologists
• Who was studied: 92 adult applicants for asbestosis compensation in the Netherlands, out of 110 screened; 18 were excluded for insufficient asbestos exposure. 91 of the 92 were male, and the median age was 75.5. The methods give the inclusion period as September 2020 to July 2022, while the results give July 2020 to August 2022.
• Exposure: Asbestos exposure was an eligibility requirement; no fiber types, products, jobs or doses are reported. The test studied was an AI procedure combining an AI classification of CT scans with the diffusing capacity of the lung for carbon monoxide (DLCO).
• Main results:
  • Full AI-driven procedure, including the extra reviewers for uncertain cases, in the 92 applicants: sensitivity 0.86 (95% CI 0.77 to 0.95), specificity 0.85 (0.76 to 0.97), accuracy 0.87 (0.79 to 0.93), ROC-AUC 0.92 (0.84 to 0.97), PR-AUC 0.95 (0.89 to 0.99), PPV 0.91 (0.85 to 0.98), NPV 0.78 (0.64 to 0.91)
  • The predefined sensitivity target was 98%, and the AI-driven procedure did not meet it
  • On the AI score alone, 8 of 92 applicants (8.7%) scored below 35 and were deemed negative, and 22 of 92 (23.9%) scored 66 or more and were assessed positive
  • 62 applicants (67.4%) had an uncertain AI score (35 to 66) and were reviewed by two additional pulmonologists
  • Without the additional reviewers, the AI score alone had a sensitivity of 0.86 (95% CI 0.78 to 0.94); adding reviewers gave a statistically significant increase in ROC-AUC (p = 0.021)
  • Agreement among the three reference pulmonologists (Randolph kappa) was 0.59 (95% CI 0.46 to 0.71) in 91 applicants; one case was excluded because a reviewer's verdict was missing
  • In the 61 applicants with an uncertain AI score, agreement among all five reviewers was 0.42 (0.30 to 0.55)
• The authors' conclusion: The authors concluded that the AI-driven index test did not meet the predefined criteria for successful validation.
• Limitations the authors state:
  • The tool was validated only in the Dutch population, and it may not apply in other countries
  • A one-sided statistical test was used
  • The team that developed the original AI tool was partly involved in this study, which may have led to bias
  • The ground truth was the verdict of three reviewers rather than a definitive diagnosis

Topics

The National Library of Medicine indexes this study under subjects our site covers: Asbestosis.

Medical terms in this study, explained in our Lexicon: Pneumoconiosis, Pulmonary Fibrosis, Asbestos Exposure, Helsinki Criteria, Exposure History, confidence interval, Fibrosis.

Related studies

• Artificial intelligence-based diagnosis of asbestosis: analysis of a database with applicants for asbestosis state aid — K.B.W. Groot Lipman, C.J. de Gooijer, et al. (2023)
• Compensation, radiographic changes, and survival in applicants for asbestosis compensation — W.O. Cookson, A.W. Musk, et al. (1985)
• Predicting Mesothelioma Using Artificial Intelligence: A Scoping Review of Common Models and Applications — M. Ram, M.R. Afrash, et al. (2025)
• Development of rapid and highly accurate method to measure concentration of fibers in atmosphere using artificial intelligence and scanning electron microscopy — Y. Iida, K. Watanabe, et al. (2021)
• Application of artificial intelligence in the analysis of asbestos fibers — R. Lee, D. Van Orden, et al. (2025)
• Rapid fiber-detection technique by artificial intelligence in phase-contrast microscope images of simulated atmospheric samples — T. Yamamoto, K. Iwasaki, et al. (2024)

Browse return to medical research.