Reference RAG on Windows, Provided Your GPU Is on the List

What it does

ChatRTX is a demo application and API layer that lets you query a local large language model grounded in your own files—text, PDFs, Word docs, and images—using retrieval-augmented generation. It bundles an inference pipeline built on TensorRT-LLM, LlamaIndex, and FAISS, wrapped in an Electron and React frontend. Everything stays on-device, and it supports voice input via Whisper and RIVA Parakeet models.

The interesting bit

The project is essentially a blueprint for NVIDIA-centric local AI: it ties together NIM microservices, CLIP for image understanding, and a vector store, but the hardware compatibility matrix reads like a nightclub guest list. Model eligibility depends on exact GPU generation and VRAM thresholds—Llama 2 13B and Gemma 7B demand 16 GB, while smaller models settle for 8 GB.

Key highlights

• Runs fully offline on Windows 11 with recent RTX GPUs (30-series and up, or RTX 6000 Ada).
• Supports multimodal RAG across text, PDF, DOCX, XML, PNG, JPG, and BMP.
• Includes both a reusable ChatRTX_APIs layer and a demo Electron/React frontend.
• Voice input handled by Whisper Medium and RIVA Parakeet NIM.
• Uses FAISS for vector search and LlamaIndex for orchestration.

Caveats

• Explicitly deprecated as of January 21, 2026, and no longer maintained.
• Windows 11 23H2/24H2 and driver 572.16 or later are mandatory; Linux and macOS need not apply.
• GPU requirements are strict and vary by model—check the matrix before assuming your card qualifies.

Verdict

Worth a look if you’re building Windows-native local RAG tools and want to see how NVIDIA stitches together TensorRT-LLM, NIM, and LlamaIndex. Everyone else—especially anyone without a recent high-VRAM RTX card—should treat it as an architecture curiosity rather than a starting point.