A Llama-2 that went to medical school (and still isn't a doctor)

What it does

EPFL’s Meditron project releases Meditron-7B and Meditron-70B, two open-weights foundation models created by continuing pre-training on Llama-2 using a curated medical corpus called GAP-Replay. The corpus blends 46,000 clinical practice guidelines, millions of PubMed abstracts and full-text articles, and a replay of general-domain tokens to preserve baseline capabilities. The output is a static model with medical knowledge baked in, intended for researchers to fine-tune or benchmark rather than deploy.

The interesting bit

The authors are refreshingly blunt about the gap between benchmark performance and clinical reality: although the finetuned 70B variant outperforms Llama-2-70B, GPT-3.5, and Flan-PaLM on medical reasoning tasks, they explicitly advise against using it in medical applications without extensive alignment and randomized controlled trials. It is a research artifact wrapped in the cautionary tape of real-world medicine.

Key highlights

• Continued pretraining from Llama-2 on a 48.1B-token corpus (GAP-Replay) that includes a newly curated set of 46K international clinical guidelines.
• Meditron-70B reportedly beats Llama-2-70B, GPT-3.5, and Flan-PaLM on multiple medical reasoning benchmarks, per the project’s paper.
• Released as static foundation models under the Llama 2 Community License (weights) and Apache 2.0 (code), with a knowledge cutoff of August 2023.
• Full training transparency: the repo documents a three-way parallelism scheme across 16 nodes of A100s using the Megatron-LLM framework, plus scripts to rebuild the training data from scratch.

Caveats

• Explicitly not instruction-tuned or safety-aligned for clinical use; the authors warn against production deployment or any use case that might impact people.
• Weights are governed by the Llama 2 Community License, which carries commercial restrictions that may complicate enterprise adoption.

Verdict

Worth a look if you are a researcher exploring domain-adaptive pretraining or medical QA benchmarks. Avoid if you need a drop-in clinical copilot—the authors have already written the disclaimer for you.