ChatGPT could score at or around the approximately 60 per cent passing threshold for the United States Medical Licensing Exam (USMLE), with responses that made coherent, internal sense and contained frequent insights, according to a new study.
Tiffany Kung and colleagues at AnsibleHealth, California, US, tested ChatGPT's performance on the USMLE, a highly standardized and regulated series of three exams, including Steps 1, 2CK, and 3, required for medical licensure in the US, the study said.
Taken by medical students and physicians-in-training, the USMLE assesses knowledge spanning most medical disciplines, ranging from biochemistry, to diagnostic reasoning, to bioethics.
After screening to remove image-based questions from the USMLE, the authors tested the software on 350 of the 376 public questions available from the June 2022 USMLE release, the study said.
The authors found that after indeterminate responses were removed, ChatGPT had scored between 52.4 per cent and 75 per cent across the three USMLE exams, the study published in the journal PLOS Digital Health said.
The passing threshold each year is approximately 60 per cent.
ChatGPT is a new artificial intelligence (AI) system, known as a large language model (LLM), designed to generate human-like writing by predicting upcoming word sequences.
Unlike most chatbots, ChatGPT cannot search the internet, the study said.
Instead, it generates text using word relationships predicted by its internal processes, the study said.
According to the study, ChatGPT also demonstrated 94.6 per cent concordance across all its responses and produced at least one significant insight, something that was new, non-obvious, and clinically valid, for 88.9 per cent of its responses.
ChatGPT also exceeded the performance of PubMedGPT, a counterpart model trained exclusively on biomedical domain literature, which scored 50.8 per cent on an older dataset of USMLE-style questions, the study said.
While the relatively small input size restricted the depth and range of analyses, the authors noted that their findings provided a glimpse into ChatGPT's potential to enhance medical education, and eventually, clinical practice.
For example, they added, clinicians at AnsibleHealth already use ChatGPT to rewrite jargon-heavy reports for easier patient comprehension.
"Reaching the passing score for this notoriously difficult expert exam, and doing so without any human reinforcement, marks a notable milestone in clinical AI maturation," said the authors.
Kung added that ChatGPT's role in this research went beyond being the study subject.
"ChatGPT contributed substantially to the writing of [our] manuscript... We interacted with ChatGPT much like a colleague, asking it to synthesize, simplify, and offer counterpoints to drafts in progress... All of the co-authors valued ChatGPT's input.
(Only the headline and picture of this report may have been reworked by the Business Standard staff; the rest of the content is auto-generated from a syndicated feed.)
You’ve hit your limit of {{free_limit}} free articles this month.
Subscribe now for unlimited access.
Already subscribed? Log in
Subscribe to read the full story →
Quarterly Starter
₹900
3 Months
₹300/Month
Smart Essential
₹2,700
1 Year
₹225/Month
Super Saver
₹3,900
2 Years
₹162/Month
Renews automatically, cancel anytime
Here’s what’s included in our digital subscription plans
Access to Exclusive Premium Stories Online
Over 30 behind the paywall stories daily, handpicked by our editors for subscribers


Complimentary Access to The New York Times
News, Games, Cooking, Audio, Wirecutter & The Athletic
Business Standard Epaper
Digital replica of our daily newspaper — with options to read, save, and share


Curated Newsletters
Insights on markets, finance, politics, tech, and more delivered to your inbox
Market Analysis & Investment Insights
In-depth market analysis & insights with access to The Smart Investor


Archives
Repository of articles and publications dating back to 1997
Ad-free Reading
Uninterrupted reading experience with no advertisements


Seamless Access Across All Devices
Access Business Standard across devices — mobile, tablet, or PC, via web or app