A Flask-like microframework for LLM tool-calling

What it does

kani is a Python microframework for building chat-based applications around language models. It handles the mechanical bits—token counting, chat history trimming, and async dispatch—while exposing function calling through a simple decorator. The model-agnostic core lets you swap backends like OpenAI, Anthropic, local Hugging Face weights, or llama.cpp without rewriting your application logic.

The interesting bit

The authors explicitly position it as the Flask to LangChain’s Django: minimal, unopinionated, and transparent. There are no hidden system prompts or under-the-hood prompt rewrites; what you write is what the model sees. That transparency, paired with first-class async and streaming support, makes it notably researcher-friendly—it even earned a paper at EMNLP 2023.

Key highlights

• @ai_function decorator exposes Python methods to the model with automatic parameter validation and retry on hallucinations.
• Streaming responses work token-by-token, even during multi-step function calls.
• Includes a zero-setup CLI for quick terminal testing against any configured backend.
• Optional multimodal support via a separate extension for image, audio, and video inputs.
• Async-by-design architecture scales to parallel chat sessions without process juggling.

Verdict

Reach for kani if you need a lightweight, hackable core for LLM agents or research prototypes where prompt transparency matters. Skip it if you are looking for a fully integrated no-code pipeline or a batteries-included orchestration platform.