Changes in artificial intelligence technology are occurring at a fast pace and transforming many industries, including healthcare. In a recent study, researchers found that ChatGPT, an AI chatbot, could potentially aid healthcare professionals in their decision-making processes. The study, led by investigators from Mass General Brigham, revealed that ChatGPT was nearly 72% accurate across all medical specialties and phases of clinical care, and 77% accurate in making final diagnoses. The findings highlight the potential for generative AI to increase access and efficiency in healthcare.
The large-language model (LLM) artificial intelligence chatbot performed equally well in both primary care and emergency settings across all medical specialties. The study, published in the Journal of Medical Internet Research, demonstrated that ChatGPT’s performance was comparable to that of a medical school graduate, such as an intern or resident. This suggests that LLMs have the potential to be valuable tools in medical practice, supporting clinical decision-making with impressive accuracy.
The researchers tested the hypothesis that ChatGPT could guide an entire clinical encounter with a patient, including recommending a diagnostic workup, deciding the clinical management course, and ultimately making the final diagnosis. They evaluated ChatGPT’s accuracy on differential diagnosis, diagnostic testing, final diagnosis, and management in a structured blinded process. Linear regressions were then used to assess the relationship between ChatGPT’s performance and the demographic information of the patient scenario.
While ChatGPT demonstrated high accuracy in several areas, there were areas where it struggled. It was only 60% accurate in making differential diagnoses, and 68% accurate in clinical management decisions. Despite these limitations, ChatGPT’s overall performance was steady across both primary and emergency care, and it did not show any gender bias in its answers.
According to Marc Succi, Associate Chair of innovation and Commercialisation and Strategic Innovation leader at Mass General Brigham, ChatGPT’s lower performance in making differential diagnoses indicates that physicians add the most value in the early stages of patient care when little information is available and a list of possible diagnoses is needed. However, Succi and his team emphasize that more benchmark research and regulatory guidance are necessary before tools like ChatGPT can be integrated into clinical care.
The potential of AI, particularly in the field of healthcare, is immense. While the study demonstrates promising results for ChatGPT, it is crucial to approach AI integration with caution and ensure that it complements the knowledge and expertise of healthcare professionals. As AI continues to evolve, it is expected to play a significant role in enhancing healthcare delivery, accessibility, and efficiency.
