Dr. Nicholas Gavin, an emergency medicine physician at Mount Sinai in New York City, was working a night shift last summer when a patient arrived with a series of disconcerting symptoms. Within seconds, his three younger colleagues (two medical students and a resident) were consulting a free, AI-powered app for doctors, OpenEvidence.
Dr. Gavin soon discovered that they were far from outliers. One-third of Mount Sinai’s 9,000 doctors were already regular users of OpenEvidence, health system executives discovered in a meeting last year with the startup’s leaders.
“That was a wonderful moment for our leadership,” said Dr. Gavin, who is also the system’s chief clinical innovation officer.
OpenEvidence’s AI app, essentially a chatbot for medicine, has become a viral hit among doctors. Talk to a doctor and they’ll likely use the app to ask specific medical questions or exchange ideas in a diagnostic dialogue.
More than half of the country’s doctors are regular users. Last month, they used it for 30 million questions and queries, almost double the number six months earlier, according to the startup. A separate survey of 1,000 doctors last year found that 45 percent of them used the app, nearly triple the percentage using ChatGPT, according to Offcall, a career information service for doctors.
That growth propelled the startup to a valuation of $12 billion in January, up from $3.5 billion last July.
But the app’s rapid adoption by doctors since its introduction in 2024 — one of the few AI-enhanced programs on the market seeking to win over doctors — has raised concerns about how and when the technology should be used in life-or-death situations. In a high-stakes field like medicine, healthcare systems are navigating thorny issues of privacy, security, and patient trust, as well as the limitations of the technology itself.
“It’s not an oracle, it’s a tool,” said Daniel Nadler, founder and CEO of OpenEvidence. “Knowledge and knowledge workers are still important.”
The medical office has been the subject of computer-aided decision making for decades, with very limited success until recent advances in AI.
The first wave of AI in medicine focused on alleviating the heavy documentation burden that contributes to physician burnout with transcriptions and summaries of patient visits, called AI writing software. The second wave, which is just beginning, aims to use AI to assist doctors with reliable information and advice to guide diagnosis and treatment while at the patient’s bedside.
Competition has intensified in recent months. UpToDate, a popular legacy e-reference for doctors, has revamped its service using AI with a chatbot interface. Doximity, an online professional network for doctors, bought an artificial intelligence startup that mines medical literature and generates summaries. Abridge, a fast-growing AI scribe maker, is adding decision support tools. And last month, OpenAI introduced ChatGPT for doctors.
OpenEvidence became a pioneer in part because it exclusively used medical journals and other high-quality research as data to train its AI models. Doctors can ask specific questions to the app or enter a patient’s characteristics and symptoms and ask for possible explanations. The app complies with federal law protecting patient health information and doctors are asked not to enter any personally identifiable information.
OpenEvidence responds with a summary of the most likely diagnoses and then offers other “more important diagnoses not to miss.” Each has links to the research articles that inform the summaries.
“AI is solving some of the problems that have long plagued the practice of medicine,” said Dr. Raja-Elie Abdulnour, director of clinical innovation at NEJM Group, which publishes The New England Journal of Medicine. “These tools just didn’t exist before and that’s why people are so excited about them now.”
However, medical experts agree that initial enthusiasm should be tempered with a large dose of caution. So far, research on the benefits and drawbacks of AI in medicine is decidedly mixed.
AI has passed standard licensing exams and outperformed human doctors in diagnosing certain cases. But AI has also stumbled, failing to accurately summarize research work or giving wrong answers to diagnostic questions. And it won’t replace humans anytime soon.
“The potential of AI is huge, but we’re not there yet,” said Dr. Eric Topol, a cardiologist and executive vice president at Scripps Research in San Diego. “It hasn’t really been tested or proven in the messy, real world of medicine.”
Dr. Topol is co-author of a recent paper, “The Illusion of Readiness in Health AI,” which found “significant competency gaps” in the capability of large AI systems when applied to healthcare.
Until now, evaluations have largely focused on the performance of so-called big language models from big tech companies like OpenAI and Google, which rely on data across the open Internet.
OpenEvidence, founded in 2022, took a more focused approach. He bet that smaller AI software models trained with highly specialized data could outperform giant models in a specific, information-rich field like medicine. The startup initially trained its software on publicly available medical data from sources such as the government’s National Library of Medicine.
The company then closed content licensing deals with The New England Journal of Medicine, The Journal of the American Medical Association, and other publishers of peer-reviewed medical literature.
OpenEvidence is available to any government-verified doctor in the United States as a free downloadable app.
“We treat doctors like consumers,” Nadler said. Users are presented with ads, many of them from pharmaceutical companies, during the roughly five seconds they wait for the AI to respond. Doctors receive ads on only 5 percent of their questions, the company said.
Bypassing the traditional gatekeepers of hospital technology departments has posed some problems. OpenEvidence has built on workplace behavior known as “shadow AI,” in which workers use such tools without the knowledge or supervision of their employers.
Some health systems are now focusing on bringing OpenEvidence into the institutional fold. Mount Sinai announced in March that it would provide a link to OpenEvidence directly from a patient’s electronic health record.
But the deal does not give the startup access to the medical center’s patient data. That integration could come later, Dr. Gavin said, but only after rigorous testing and monitoring.
Protecting patient privacy and security will be “top of mind,” he said, adding that “we’re not going to just throw a patient’s data at a private company.”
Doctors in smaller practices across the country, especially in rural areas, say they have been won over by technology.
In Corinth, Mississippi, Dr. Ben Long considers himself an AI skeptic. But he assured her that OpenEvidence generates answers based only on high-quality, peer-reviewed information.
At first, Dr. Long used it primarily as a reference tool, asking objective questions. But now he sees the app more as “a consultant, a thought partner” with whom he dialogues, he said.
“AI forces you to think more deeply about your own thinking, challenging your assumptions and why you might be wrong,” Dr. Long said.
AI can also allow doctors to tap into knowledge that would normally be the province of specialists.
Dr. Barbara Creighton often diagnoses and treats complex cases at a community hospital in Fairbanks, Alaska. They can involve multiple conditions and failing organs. In a large medical center, a team of specialists may be consulted: an infectious disease expert, a pulmonologist, and a gastroenterologist, for example.
Dr. Creighton’s small hospital is not as staffed. You have an agreement with a large medical center to pay for specialized consulting sessions. It now increasingly relies on OpenEvidence to answer many questions, saving the hospital time and money.
“It’s like having a bunch of specialists in your pocket,” Dr. Creighton said.
At Mount Sinai, Dr. Gavin said he saw AI technology as a powerful tool to help realize the promise of precision medicine with treatments tailored to people.
Progress will require a “mosaic of solutions” from hospitals, medical schools and private companies, he said. It remains to be seen whether OpenEvidence thrives and plays a role in that long-term future.
“But it represents a step in that direction,” Dr. Gavin said.
