A new study is raising fresh questions about how far AI tools can go in health care. The research shows that chatbots often fail when asked to make early-stage medical calls, especially when user input is thin or unclear.
Meet Samuel – Your Personal Investing Prophet
- Start a conversation with TipRanks’ trusted, data-backed investment intelligence
- Ask Samuel about stocks, your portfolio, or the market and get instant, personalized insights in seconds
The study, published in Jama Network Open, tested 21 large language models from firms such as OpenAI, Alphabet (GOOGL), Anthropic, xAI, and DeepSeek. It found that failure rates topped 80% when the models had to suggest a range of possible causes with limited data.
In contrast, once full patient data was shared, accuracy rose fast. In those cases, the best models reached more than 90% accuracy, and failure rates fell below 40%.
As Arya Rao, the lead author, said, “These models are great at naming a final diagnosis once the data is complete, but they struggle at the open-ended start of a case, when there isn’t much information.”
Implications for Big Tech and AI Strategy
This gap matters to firms racing to build AI into everyday tools. Companies like OpenAI, Anthropic, and Alphabet are pushing chatbots into search, apps, and user help flows. However, this study shows that these systems still lack strong early judgment, which is key in real-world use.
At the same time, firms are aware of the risk. For example, Google said its Gemini tool prompts users to check facts, while Anthropic noted that its Claude system guides users to seek expert care. OpenAI also states that its tools are not meant to replace licensed medical advice.
Still, this could shape how investors view the next phase of AI growth. While tools from firms like Alphabet and OpenAI show strong gains in speed and scale, their limits in judgment may slow use in high-risk fields like health care.
That said, there is still upside. Experts note that AI could help in areas with low access to care. As one researcher said, these tools “may have a role to play, particularly in situations or geographies in which access to doctors is limited.”
In the end, the study does not undercut the value of AI. Instead, it shows where the tech works best today and where firms still need to improve.
Using TipRanks’ Comparison Tool, we’ve compared notable companies that employ chatbots, such as Claude by Anthropic and OpenAI’s ChatGPT. The comparison tool helps investors gain a broader outlook on each stock and the industry as a whole.
