Inter-observer agreement and accuracy in classifying radiographs for pneumoconiosis among Asian physicians taking AIR Pneumo certification examination

Our plain-English summary of Inter-observer agreement and accuracy in classifying radiographs for pneumoconiosis among Asian physicians taking AIR Pneumo certification examination, by N.A. J-P, A.D. Susanto, E. Samoedro, et al (Industrial Health, 2022).

The study

• Authors: N.A. J-P, A.D. Susanto, E. Samoedro, M. Mansyur, S. Tungsagunwattana, S. Lertrojanapunya, P. Subhannachart, S. Siriruttanapruk, N. Dumavibhat, E. Algranti, J.E. Parker, K.G. Hering, H. Kanayama, T. Tamura, Y. Kusaka, N. Suganuma
• Journal: Industrial Health
• Published: 2022;60(5):459-469
• Cited by: 7 articles, as counted by NIH iCite
• Read it: PubMed · Publisher · Free full text

Abstract

This study examined inter-observer agreement and diagnostic accuracy in classifying radiographs for pneumoconiosis among Asian physicians taking the AIR Pneumo examination. We compared agreement and diagnostic accuracy for parenchymal and pleural lesions across residing countries, specialty training, and work experience using data on 93 physicians. Physicians demonstrated fair to good agreement with kappa values 0.30 (95% CI: 0.20-0.40), 0.29 (95% CI: 0.23-0.36), 0.59 (95% CI: 0.52-0.67), and 0.65 (95% CI: 0.55-0.74) in classifying pleural plaques, small opacity shapes, small opacity profusion, and large opacities, respectively. Kappa values among Asian countries ranging from 0.25 to 0.55 (pleural plaques), 0.47 to 0.73 (small opacity profusion), and 0.55 to 0.69 (large opacity size). The median Youden's J index (interquartile range) for classifying pleural plaque, small opacity, and large opacity was 61.1 (25.5), 76.8 (29.3), and 88.9 (23.3), respectively. Radiologists and recent graduates showed superior performance than other groups regarding agreement and accuracy in classifying all types of lesions. In conclusion, Asian physicians taking the AIR Pneumo examination were better at classifying parenchymal lesions than pleural plaques using the ILO classification. The degree of agreement and accuracy was different among countries and was associated with background specialty training.

In plain English

This study measured how consistently and accurately Asian physicians classified chest X-rays for pneumoconiosis (dust-related lung disease) and pleural plaques. The authors analysed 5,580 readings by 93 physicians taking the AIR Pneumo examination in Thailand and Indonesia. All read the same 60 X-rays, diagnosed by 12 NIOSH-certified B Readers. Agreement between physicians (kappa) was 0.30 (95% CI 0.20 to 0.40) for pleural plaques, 0.29 (0.23 to 0.36) for small opacity shape, 0.59 (0.52 to 0.67) for small opacity profusion and 0.65 (0.55 to 0.74) for large opacities. The authors called this fair to good. Accuracy scores for detecting pleural plaques, small opacities and large opacities were not percentages of X-rays read correctly. They were Youden's J index (sensitivity plus specificity minus 1) times 100. Median scores (interquartile ranges) were 61.1 (25.5), 76.8 (29.3) and 88.9 (23.3) respectively. Radiologists, Country 4 physicians and recent graduates (five or fewer years since graduating) had the highest agreement and accuracy. The authors concluded these physicians were better at classifying parenchymal (lung tissue) lesions than pleural plaques. They noted exam reading may differ from routine work.

We wrote this plain-English summary from the full article with the help of AI, and a second AI model checked each sentence against the article. It is not the authors' wording; the abstract and the article are their own account.

What the study did and found

• Design: Inter-observer agreement and diagnostic accuracy study using physicians' readings from certification examinations, compared with an expert panel's diagnoses
• Who was studied: 93 physicians residing in India, Indonesia, Malaysia and Thailand who took the AIR Pneumo examination after a 2-day training course. Their specialties were pulmonology (40), occupational medicine (25), radiology (15), public health (4) and missing (9). Eighteen percent had never seen a pneumoconiosis chest radiograph.
• Exposure: Not a study of asbestos or dust exposure. Physicians classified a 60-radiograph examination set whose diagnoses were set by 12 NIOSH-certified B Readers. Nine films had pleural plaques and 31 had small opacities. Of those 31, 9 also had large opacities. Physicians were grouped by residing country, specialty, years after graduation and number of pneumoconiosis radiographs previously reviewed.
• Main results:
  • Across all physicians, kappa agreement was 0.30 (95% CI: 0.20–0.40) for pleural plaques and 0.29 (95% CI: 0.23–0.36) for small opacity shapes. It was 0.59 (95% CI: 0.52–0.67) for small opacity profusion and 0.65 (95% CI: 0.55–0.74) for large opacities. The authors described this as fair to good agreement.
  • Between the four countries, kappa for the presence or absence of pleural plaques ranged from 0.25 to 0.55 (fair to moderate agreement).
  • Across all 93 physicians, median accuracy scores (interquartile range) were 61.1 (25.5) for pleural plaque, 76.8 (29.3) for small opacity and 88.9 (23.3) for large opacity. These scores are Youden's J index (sensitivity plus specificity minus 1) multiplied by 100. The small opacity shape score was different: it was percent agreement with the experts.
  • Three groups showed the highest accuracy: physicians from Country 4, those with radiology training, and those five or fewer years after graduation. These groups also had the highest agreement for all lesion types.
  • No substantial difference in accuracy was found between groups formed by how many pneumoconiosis chest radiographs physicians had reviewed before.
  • Of the 40 radiographs with small opacities, the expert panel itself agreed on small opacity shape in only 24.
• The authors' conclusion: The authors concluded that Asian physicians taking the AIR Pneumo examination were better at classifying parenchymal lesions than pleural plaques. They found that agreement differed among countries and was associated with specialty training. They suggested that specific ILO classification training and continuing practice would improve accuracy.
• Limitations the authors state:
  • The data came from examinations, where participants might expect more abnormal radiographs and read differently than in routine work.
  • Because the physicians share an interest in pneumoconiosis, the findings may not represent Asian physicians in general.
  • The authors had no information on specialty training requirements in each country.
  • Specialties were unevenly represented within each country, so results for individual countries need careful interpretation.
  • The groups had different numbers of readers, which might affect the kappa estimates.

Topics

The National Library of Medicine indexes this study under subjects our site covers: Pleural plaques and thickening.

Medical terms in this study, explained in our Lexicon: Pneumoconiosis, Pleural Plaques, ILO Classification.

Related studies

• Asian Intensive Reader of Pneumoconiosis program: examination for certification during 2008-2020 — N.A. J-P, A.D. Susanto, et al. (2024)
• Assessment of physicians' proficiency in reading chest radiographs for pneumoconiosis, based on a 60-film examination set with two factors constituting eight indices — T. Tamura, Y. Kusaka, et al. (2018)
• Radiographic diagnosis of Pneumoconioses by AIR Pneumo-trained physicians: Comparison with low-dose thin-slice computed tomography — S. Nogami, N.A. J-P, et al. (2020)
• Effect of a two-hour training on physicians' skill in interpreting Pneumoconiotic chest radiographs — N.R. Ngatu, S. Suzuki, et al. (2010)
• Occupational lung disease survey of respiratory physicians in Northern Ireland — T.F. McKeagney, K. Addley, et al. (2015)
• Inter-reader agreement in HRCT detection of pleural plaques and asbestosis in participants with previous occupational exposure to asbestos — F. Laurent, C. Paris, et al. (2014)

Browse more research in Industrial Health, or return to medical research.