Medical AI that routes X-rays through a board of agents

What it does The Multi-Agent Medical Assistant is a chatbot that hands off medical queries to a small team of agents: a computer-vision model looks at chest X-rays and skin lesions, a RAG agent searches embedded medical documents in Qdrant, and a web-search agent fetches recent papers when confidence is low. A human-in-the-loop step vets AI-generated imaging diagnoses before they are returned to the user.

The interesting bit Most demo RAG projects stop at vector search; this one treats retrieval as an assembly line. Docling parses PDFs into markdown, an LLM semantically chunks the text and tables, queries get expanded with medical synonyms, and Qdrant runs hybrid BM25-plus-dense search reranked by a HuggingFace cross-encoder. If the pipeline’s log-probability score still looks shaky, control passes to a web-search agent rather than letting the model hallucinate.

Key highlights

• Multi-agent orchestration via LangGraph with confidence-based routing between RAG and web search.
• Hybrid retrieval (BM25 + dense vectors) with cross-encoder reranking and explicit links to source documents and images.
• Computer vision modules for chest X-ray classification and skin lesion segmentation, both built in PyTorch.
• Human-in-the-loop validation gate for imaging diagnoses.
• Voice I/O through the Eleven Labs API.

Caveats

• Brain tumor detection is marked “TBD” and listed as an upcoming feature, so the imaging suite is not yet complete.
• The codebase is currently wired for Azure OpenAI (gpt-4o, text-embedding-ada-002); the README notes that using other LLMs or local models requires code changes throughout the agents directory.
• It relies on several external paid APIs (OpenAI, Tavily, Eleven Labs) and is designed to run in Docker, so it is not an offline-first tool.

Verdict Worth exploring if you are learning how to wire LangGraph, guardrails, and hybrid RAG into a production-shaped prototype; look elsewhere if you need a fully offline, turnkey clinical diagnostic system.