What it does Kiln is a free desktop app and open-source Python library for building AI systems end-to-end. You define a task once, then run evals, optimize prompts, spin up RAG, generate synthetic data, fine-tune models (60+ across Fireworks, Together, Vertex), and compose multi-agent hierarchies—all against the same dataset. The app runs locally with your own API keys or fully offline via Ollama; teammates sync through Git without needing to know what git rebase means.

The interesting bit The “Auto-Optimize” feature doesn’t just score your prompt—it mutates prompts, swaps models, and tunes parameters across hundreds of combinations to find what actually works for your eval dimensions. That’s the tedious part most tools skip.

Key highlights

• Desktop apps for macOS, Windows, and Linux; same MIT-licensed kiln-ai library ships to production
• 190+ pre-tested models across OpenAI, Anthropic, Gemini, Bedrock, Ollama, OpenRouter, Groq, and any OpenAI-compatible endpoint
• Drag-and-drop RAG from PDFs, images, video, audio with auto-generated evals from your own documents
• Zero-code fine-tuning with serverless deployment included
• Git-native collaboration: PMs and SMEs rate outputs in the GUI, engineers deploy the same tasks via Python or REST API

Caveats

• The AI Assistant, Auto-Optimize, and Eval Builder are gated behind Kiln Pro; core app remains functional without them
• Desktop app is “source-available” under fair-code, not MIT—only the core library and REST server are fully open source

Verdict Worth a look if you’re tired of stitching together six separate tools or feeding proprietary data to SaaS platforms. Skip it if you need a pure code-first framework without any GUI baggage, or if you want everything fully open source.