Real-time voice search that cites its sources

What it does This sample implements the VoiceRAG pattern: a browser-based voice interface streams audio to a Python backend, which calls the Azure OpenAI GPT-4o Realtime API. When you ask a question, the backend retrieves relevant documents from Azure AI Search and feeds them to the model so the spoken answer is grounded in your data. The app also displays citations for the search results it used.

The interesting bit Instead of chaining separate speech-to-text and text-to-speech services, the app uses GPT-4o’s native audio capabilities end-to-end. The backend’s RTMiddleTier orchestrates the realtime websocket connection and injects Azure AI Search results directly into the conversation, letting the model answer with a voice that actually knows your documents.

Key highlights

• End-to-end voice: browser microphone input and audio output via the GPT-4o Realtime API.
• RAG with citations: retrieves documents from Azure AI Search and shows which sources were used to generate the answer.
• Azure-native deployment: includes infrastructure-as-code and a Dockerfile for Azure Container Apps, with Managed Identity for authentication.
• Local development supported: can point the backend at existing Azure OpenAI and AI Search instances.
• Synthetic sample data included for demonstration (Contoso electronics policies).

Caveats

• Deployment incurs immediate Azure costs, primarily from AI Search, and charges may accrue even if you interrupt the provisioning command.
• The list of available Azure OpenAI deployment regions is tied to model availability and may become outdated.
• This is explicitly a demonstration sample, not a production-hardened framework.

Verdict Worth a look if you are building voice-first Azure AI apps and need a working reference for realtime audio plus retrieval. Skip it if you want a cloud-agnostic solution or a finished product without Azure lock-in.