Med-PaLM — A Language Model Clears the Pass Mark on Licensing-Exam Questions

Med-PaLM is a medical large language model built by researchers at Google Research and DeepMind. The paper, "Large Language Models Encode Clinical Knowledge", was posted to arXiv on 26 December 2022 and published in Nature on 12 July 2023. Because it cleared what Google calls the pass mark, 60%, on MedQA, a collection of US Medical Licensing Examination (USMLE)-style questions, it is often described as the first AI to pass the medical licensing exam. Google's own wording is that Med-PaLM was the first AI system to surpass the pass mark (>60%) on USMLE-style questions. It did not sit the real exam; it was scored on a question set in the exam's format.

The paper appeared less than a month after the public launch of ChatGPT, when the question of whether language models could be used in medicine suddenly felt real. Alongside its scores, the paper's answer to that question was: not yet at the level of clinicians.

What the team did

The starting point was Google's 540-billion-parameter language model PaLM and Flan-PaLM, a version tuned on a large set of instructions. The team first assembled MultiMedQA, a benchmark for medical question answering. It combined six existing datasets, including the USMLE-style MedQA, MedMCQA (questions from Indian medical entrance exams), PubMedQA (yes/no/maybe questions answered from a PubMed abstract) and the clinical topics of MMLU, with a new dataset, HealthSearchQA, of 3,173 commonly searched consumer health questions.

Combining several prompting strategies, Flan-PaLM set the state of the art on every multiple-choice dataset, reaching 67.6% on MedQA, more than 17 points above the previous best. But when clinicians read its long-form answers, serious gaps appeared. The team therefore introduced instruction prompt tuning, a parameter-efficient way of adapting a model to a new domain using a few exemplars crafted with clinicians. Rather than retraining the model, it is a light form of tuning; the result was Med-PaLM.

How physicians rated it

According to the Nature version, a panel of nine clinicians based in the US, UK and India rated answers to 140 consumer medical questions without knowing their source, with each answer rated by one clinician.

On these two measures Med-PaLM was close to clinicians. The paper's conclusion was nonetheless that Med-PaLM "performs encouragingly, but remains inferior to clinicians." Lay raters likewise found Med-PaLM's answers more helpful and better aligned with the question's intent than Flan-PaLM's, but still behind clinicians'.

Med-PaLM 2: 86.5%

Google introduced Med-PaLM 2 at its annual health event, The Check Up, in March 2023, and posted the paper to arXiv on 16 May. It swapped the base model for PaLM 2 and combined medical domain fine-tuning with prompting strategies including a new "ensemble refinement" approach.

The 67.2% for Med-PaLM is the figure given in the Med-PaLM 2 paper. The 67.6% in the first paper's abstract belongs to Flan-PaLM, the model Med-PaLM was built from; the two numbers describe different models.

The 2025 Nature Medicine version added a pilot study with real-world medical questions. Specialists preferred Med-PaLM 2's answers to generalist physicians' answers 65% of the time, though specialists' own answers were still preferred overall, and both specialists and generalists rated Med-PaLM 2 as safe as physicians' answers. Google later described Med-PaLM 2 as one of the research models powering MedLM, a family of healthcare models for Google Cloud customers. In July 2023 the team also published Med-PaLM M, a multimodal version handling images and genomics (pictured above).

Other models at the same time

Exam-style scores were quickly matched elsewhere. On 20 March 2023 Nori and colleagues reported that OpenAI's GPT-4, a general model not specialised for medicine, exceeded the USMLE passing score by over 20 points without specialised prompt crafting, and outperformed Med-PaLM.

For Japan's national medical licensing exam, Kasai and colleagues reported on 31 March 2023 that GPT-4 passed all six years of the exam they tested. The same paper warned that the models sometimes chose options that must be strictly avoided in Japanese practice, such as suggesting euthanasia, and that Japanese text costs more through the API and fits less into the context window because of how non-Latin scripts are tokenised.

The limitations the authors listed

The Med-PaLM paper discusses its limitations in a dedicated section. The main points:

• Multiple-choice questions are "inherently easier" than clinical work, because they are built on expert-written vignettes chosen to have a generally preferred answer; real care means eliciting information from patients and turning findings into an assessment and plan.
• Only English-language datasets were used; multilingual evaluation is a pressing need.
• The human-evaluation axes were neither exhaustive nor objective, and each answer was rated by a single person; a much larger and more diverse pool of raters is needed.
• Medical consensus changes over time and has been coloured by discrimination by race, gender, age and ability; a model may reflect the consensus of the past.
• Clinical use would require further capabilities, including grounding answers in authoritative sources, communicating uncertainty and meeting the safety requirements of medicine.

What changed

Med-PaLM showed two things at once: that a general language model could be adapted to medicine with little effort, and that exam scores were not enough. The authors themselves showed a wide gap between multiple-choice accuracy and how physicians judged the answers, and named their evaluation framework as the first key contribution of the paper.

For the public, HealthSearchQA pointed to a near future in which language models answer the health questions people type into search engines. The paper put a number on the risk, as the share of answers that could lead to harm. Yet the distance between language models and AI that is reviewed and sold as a medical device remained large. For reviewed devices see IDx-DR and EndoBRAIN; for medical AI that fell short in practice see the Epic Sepsis Model.

For language models matching top human performance in mathematics, see AI's gold-medal-level result at the International Mathematical Olympiad. The wider context is on the AI timeline and the AI and society timeline, and the labs are covered on the Google and DeepMind pages.