A radiology department packed into a Python agent

What it does

MedRAX is a LangGraph-based agent that hands off chest X-ray images to a stable of specialized computer-vision models—classifiers, segmenters, grounding tools, and report generators—and synthesizes their outputs into answers for clinical questions. It uses GPT-4o as the default coordinator, though it can be pointed at local or OpenAI-compatible endpoints if you prefer to keep your pixels in-house. A Gradio interface is included for interactive use.

The interesting bit

Rather than training yet another monolithic medical foundation model, the authors treat radiology as a routing problem: the LLM decides which off-the-shelf tool to call for detection, localization, comparison, or report generation, then stitches the results together. They also published ChestAgentBench, a 2,500-query benchmark built from 675 expert-curated cases, to measure whether the agent actually reasons or just hallucinates confidently.

Key highlights

• Modular toolset: plugs in CheXagent, LLaVA-Med, Maira-2, MedSAM (coming soon), and others, with auto-downloading weights for most tools.
• Benchmark included: ChestAgentBench spans detection, classification, localization, diagnosis, and more across 2,500 complex queries.
• Deployment flexibility: runs against OpenAI’s API or local backends via Ollama, LM Studio, and OpenAI-compatible providers like DashScope.
• Resource-conscious: supports 8-bit/4-bit quantization for heavier models and lets you initialize only the tools you need.
• DICOM support and synthetic X-ray generation via RoentGen (though RoentGen weights require manual setup).

Caveats

• MedSAM segmentation support is listed as a future update, not yet available.
• Several components (LLaVA-Med, grounding) are memory-hungry even with quantization, so “local” deployment still assumes a decent GPU.
• The default backbone is GPT-4o, meaning out-of-the-box usage sends medical images to OpenAI unless you rewire it to a local model.

Verdict

Worth a look if you’re building clinical AI pipelines and want a ready-made integration layer for existing chest X-ray models. Skip it if you need a fully offline, lightweight drop-in solution today.