ChatGPT showed some promise for relatively straightforward cardiology questions but performed less well in more complex case vignettes
ChatGPT has the potential as an AI-assisted decision-support tool for questions that are relatively straightforward. In contrast, it performed less well when providing answers to more complicated case vignettes.
Chat Generative Pre-trained Transformer (ChatGPT) is an interactive AI model. The system follows instructions and provides a detailed response. Furthermore, the system has the potential to assist with medical education and even clinical decision-making. In the present study, researchers set out to assess ChatGPT’s performance at answering cardiovascular questions and in providing suggestions in case vignettes. For the questions, the reference standard was the medical expert who developed the questions. As for the 20 vignettes, the standard was the attending physician or consulted expert and the advice provided was checked with reference to clinical guidelines. The straightforward cardiovascular questions, related to several topics including acute coronary syndrome, atrial fibrillation and cardiovascular risk management. Vignettes involved symptoms that were potentially due to a cardiac problem (e.g., chest pain, dyspnoea) or required a diagnostic/treatment plan.
ChatGPT performance
Using 50 multiple choice cardiovascular questions, ChatGPT was correct in 74% (37/50) of cases. Scoring varied from 80% (for coronary artery disease) to 60% (cardiovascular risk management). For the vignettes, when seeking primary care advice, ChatGPT correctly answered questions in 90% of cases. When asked more complicated questions, the system was correct in only 50% of cases.
The authors felt that ChatGPT performed well with straightforward, low complexity questions. However, they felt more work was needed to fully evaluate the system’s potential.
Citation
Harskamp RE et al. Performance of ChatGPT as an AI-assisted decision support tool in medicine: a proof-of-concept study for interpreting symptoms and management of common cardiac conditions (AMSTELHEART-2). MedRxiv 2023
