What it does

This is a Create React App starter that wires up your browser to Gemini’s multimodal Live API over a WebSocket. It bundles an event-emitting WebSocket client, audio I/O plumbing, and a unified log viewer so you can see the traffic flowing in both directions. Out of the box it handles microphone input, webcam capture, screen sharing, and streaming audio playback.

The interesting bit

The project treats the WebSocket not as a raw socket but as an event-emitting client with a React context wrapper, which means you can drop useLiveAPIContext() into a component and suddenly have the model calling your local functions. The included Altair example shows the pattern: define a function declaration, hand it to Gemini as a tool, and render the returned Vega graph in a useEffect.

Key highlights

• Branches with working demos: proactive audio, a weather bot, a list generator, and a code explainer
• Built-in media handling for mic, webcam, and screen capture without reaching for extra libraries
• Unified log view for debugging the back-and-forth with the model
• Free tier works: just grab a Gemini API key and npm start

Caveats

• Bootstrapped with Create React App, which is officially in maintenance mode and lacks modern tooling like Vite or Turbopack
• Explicitly labeled “an experiment, not an official Google product” with a “your mileage may vary” disclaimer

Verdict

Grab this if you want to prototype a voice-or-vision app with Gemini without writing WebSocket frame parsing from scratch. Skip it if you need a production-ready framework or if you’re allergic to Create React App.