What it does

JARVIS is a voice-controlled AI assistant for macOS. You speak to it in Chrome, it transcribes via Web Speech API, routes through a FastAPI backend, and responds with synthesized speech in a JARVIS-style British voice. It can read your Apple Calendar, Mail, and Notes; browse the web; take notes; manage tasks; remember preferences; and plan your day. The visual hook is a Three.js particle orb that reacts to the audio output.

The interesting bit

The project leans hard into the MCU fantasy but backs it with real plumbing: AppleScript bridges to native macOS apps (no OAuth dances), a 2300-line Python server that switches between Claude Haiku for speed and Claude Opus for depth, and a [ACTION:BUILD] tag system that literally spawns Claude Code subprocesses to write software while you watch. The memory layer uses SQLite with FTS5 full-text search, which is more thought than most hobby assistants put into persistence.

Key highlights

• Voice loop: Web Speech API → WebSocket → FastAPI → Claude Haiku → Fish Audio TTS → WebSocket back to browser, with a Three.js orb pulsing to the audio stream
• Native macOS integration via AppleScript for Calendar, Mail (read-only by design), Notes, and Terminal
• Action tags like [ACTION:BUILD] and [ACTION:RESEARCH] trigger Claude Code sessions or deep research with Opus
• SQLite + FTS5 memory system for cross-session preferences and facts
• Dual-speed LLM strategy: Haiku for conversation latency, Opus for complex tasks
• Screen awareness: knows what apps are open for context-aware responses

Caveats

• macOS-only; AppleScript ties it tightly to the platform, and Linux/Windows support is listed as a future contribution area
• Requires three external API keys (Anthropic, Fish Audio) plus Claude Code CLI and Chrome specifically
• Setup involves manual SSL certificate generation and running separate backend/frontend terminals
• Demo GIF in README is currently a <!-- TODO --> placeholder
• License restricts commercial use without separate arrangement

Verdict

Worth a look if you want a hackable, voice-first AI assistant with actual macOS integration and don’t mind the MCU cosplay. Skip it if you need cross-platform support, want something that works out of the box, or find the Tony Stark roleplay grating.