The clinical meaning of the area under a receiver operating characteristic curve for the evaluation of the performance of disease markers

Our plain-English summary of The clinical meaning of the area under a receiver operating characteristic curve for the evaluation of the performance of disease markers, by S. Parodi, D. Verda, F. Bagnasco, et al (Epidemiology and Health, 2022).

The study

• Authors: S. Parodi, D. Verda, F. Bagnasco, M. Muselli
• Journal: Epidemiology and Health
• Published: 2022;44:e2022088
• Cited by: 33 articles, as counted by NIH iCite
• Read it: PubMed · Publisher · Free full text

Abstract

Objectives: The area under a receiver operating characteristic (ROC) curve (AUC) is a popular measure of pure diagnostic accuracy that is independent from the proportion of diseased subjects in the analysed sample. However, its actual usefulness in the clinical context has been questioned, because it does not seem to be directly related to the actual performance of a diagnostic marker in identifying diseased and non-diseased subjects in real clinical settings. This study evaluates the relationship between the AUC and the proportion of correct classifications (global diagnostic accuracy, GDA) in relation to the shape of the corresponding ROC curves.

Methods: We demonstrate that AUC represents an upward-biased measure of GDA at an optimal accuracy cut-off for balanced groups. The magnitude of bias depends on the shape of the ROC plot and on the proportion of diseased and non-diseased subjects. In proper curves, the bias is independent from the diseased/non-diseased ratio and can be easily estimated and removed. Moreover, a comparison between 2 partial AUCs can be replaced by a more powerful test for the corresponding whole AUCs.

Results: Applications to 3 real datasets are provided: a marker for a hormone deficit in children, 2 tumour markers for malignant mesothelioma, and 2 gene expression profiles in ovarian cancer patients.

Conclusions: The AUC is a measure of accuracy with potential clinical relevance for the evaluation of disease markers. The clinical meaning of ROC parameters should always be evaluated with an analysis of the shape of the corresponding ROC curve.

In plain English

This was a statistical methods paper, not a study of asbestos exposure. The authors examined how the area under a receiver operating characteristic (ROC) curve, a common score of how well a test separates diseased from non-diseased people, relates to the share of people it classifies correctly. They derived the theory and applied it to three previously published datasets. One dataset compared two tumour markers, SMRP and Cyfra21-1, in 52 patients with malignant pleural mesothelioma and 117 patients with benign pleurisies or pleural metastases from other malignancies. In that dataset the area was 0.842 (95% confidence interval 0.774 to 0.910) for SMRP and 0.760 (0.687 to 0.832) for Cyfra21-1. The Cyfra21-1 curve departed significantly from the expected 'proper' shape (p=0.003). Comparing the markers over specificities of 0.80 to 1.00 gave p less than 0.001. Comparing the whole areas gave p=0.043. The authors concluded that the area is a useful measure with potential clinical relevance and that curve shape should always be checked. They noted ROC analysis has been criticised for evaluating risk scores.

We wrote this plain-English summary from the full article with the help of AI, and a second AI model checked each sentence against the article. It is not the authors' wording; the abstract and the article are their own account.

What the study did and found

• Design: Statistical methods paper (mathematical derivation) illustrated with re-analysis of 3 previously published datasets
• Who was studied: Three previously published datasets: 79 patients with childhood growth hormone deficit (31 with permanent deficit); 52 patients with malignant pleural mesothelioma and 117 controls (55 benign pleurisies, 62 pleural metastases from other malignancies); and normal (n=23) and malignant (n=30) ovarian tissues. The article does not state where or when the mesothelioma patients were studied.
• Exposure: No asbestos or fiber exposure was studied. In the mesothelioma dataset, what was evaluated was the diagnostic performance of two tumour markers, SMRP and Cyfra21-1.
• Main results:
  • Mesothelioma dataset: the area under the ROC curve was 0.842 (95% CI, 0.774 to 0.910) for SMRP and 0.760 (95% CI, 0.687 to 0.832) for Cyfra21-1.
  • Mesothelioma dataset: the Cyfra21-1 curve was asymmetrical and departed significantly from the theoretical proper model (p=0.003, Metz and Kronman test).
  • Mesothelioma dataset: in the specificity range 0.80 to 1.00, the difference between the partial areas of SMRP and Cyfra21-1 was 0.070 (p less than 0.001).
  • Mesothelioma dataset: the difference between the whole areas of the two markers was only marginally statistically significant (p=0.043).
  • Mesothelioma dataset: for SMRP, whose curve was consistent with the proper model, the cut-off with equal sensitivity and specificity was 8.36 nmol/L, with a diagnostic odds ratio of 11.7 and a global diagnostic accuracy of 76.3% (shown in Figure 6).
  • Growth hormone deficit dataset: the area under the ROC curve for the insulin-tolerance test was 0.922 (95% CI, 0.855 to 0.990).
  • Growth hormone deficit dataset: global diagnostic accuracy estimated by the authors' method was 86% (95% CI, 79 to 96), compared with an observed 83.5%.
  • Ovarian cancer dataset: the areas were 0.625 for gene 1 and 0.839 for gene 2. Comparing whole areas gave p=0.027, which the authors said suggested gene 2 could perform better. Comparing partial areas would have given p=0.141.
  • Theoretical result: for proper curves, the largest gap (bias) between the area and global accuracy was 0.07, at a diagnostic odds ratio of 17.3, which corresponds to an area of 0.876.
• The authors' conclusion: The authors concluded that the area under the ROC curve is a useful measure of accuracy with potential clinical relevance for evaluating disease markers. For curves of proper shape, they concluded it is an upper limit of accuracy. They also concluded that ROC parameters should always be evaluated together with an analysis of the curve's shape.
• Limitations the authors state:
  • ROC analysis has been criticised for risk scores: a clinically significant change in a person's probability of disease might not show up as an appreciable increase in the area.
  • Whether ROC analysis is useful for assessing the gain from adding a marker to a baseline predictor remains controversial.
  • When a cut-off is chosen according to the costs of false results and the expected proportion of diseased people, the clinical usefulness of the area remains to be clarified.

Topics

Medical terms in this study, explained in our Lexicon: Mesothelioma, Ovarian Cancer.

Related studies

• Megakaryocyte potentiating factor as a tumor marker of malignant pleural mesothelioma: evaluation in comparison with mesothelin — K. Iwahori, T. Osaki, et al. (2008)
• Clinical evaluation of circulating miR-548a-3p and -20a expression in malignant pleural mesothelioma patients — M. Matboli, A.E. Shafei, et al. (2018)
• Clinical evaluation of seven tumour markers in lung cancer diagnosis: can any combination improve the results? — M. Plebani, D. Basso, et al. (1995)
• Effect of protection against hot climate on growth performance, physiological response and endocrine profile of growing lambs under semi-arid tropical environment — K. De, D. Kumar, et al. (2017)
• Prospective Analysis of Mesotheliomas in Subjects With BAP1 Cancer Syndrome: Clinical Characteristics and Epigenetic Correlates of Disease — X. Wu, F.V. Hernandez, et al. (2025)
• Evaluation of seven tumour markers in pleural fluid for the diagnosis of malignant effusions — M. Miédougé, P. Rouzaud, et al. (1999)

Browse return to medical research.