From X-rays to 3D volumes: a medical MLLM family unifying modalities

What it does

The HealthGPT series hosts two medical multimodal language models in one repository. HealthGPT, accepted as an ICML 2025 Spotlight paper, unifies visual comprehension and generation through heterogeneous LoRA adapters and a task-specific router. HealthGPT-Pro, built on Qwen3-VL, advances medical understanding across 14 modalities—including CT, MRI, endoscopy, and histopathology—using roughly 13 million training samples.

The interesting bit

Most medical vision models either analyze images or generate new ones; HealthGPT handles both by routing tasks to specialized H-LoRA plugins within a single autoregressive framework. HealthGPT-Pro deliberately retains general-domain instruction data during fine-tuning, avoiding the common pitfall of medical-only tuning that strips away broader reasoning and obedience.

Key highlights

• HealthGPT supports 7 comprehension and 5 generation tasks through hierarchical visual perception and heterogeneous LoRA adapters.
• HealthGPT-Pro ingests 3D volumetric data alongside 2D images and text within one model.
• HealthGPT-Pro-8B scores highest on average across the listed medical text and multimodal benchmarks, with the 4B variant placing second in several categories.
• Weights for both model families and the VL-Health dataset are available on Hugging Face under an Apache 2.0 license.

Caveats

• While HealthGPT-Pro leads on average benchmark scores, it still trades individual category wins with competitors such as Hulu-Med-7B rather than dominating every task.
• A news entry in the README is dated 2026-05-02, which appears to be a typo for 2025.

Verdict

Medical AI researchers and multimodal model builders should bookmark this; clinicians looking for a vetted, deployment-ready diagnostic tool should wait for downstream validation.