Skip the scaffolding: a FastAPI template for stateful AI agents

What it does

This is essentially kind, well-organized glue code: a FastAPI template that wires together LangGraph, mem0, pgvector, Langfuse, and the usual production suspects into one scaffold. It bundles stateful conversation checkpointing, long-term memory, tool calling, JWT authentication, rate limiting, and structured observability. You bring the agent logic; it brings the database migrations, retry policies, and tracing hooks.

The interesting bit

The LLM service implements a circular fallback: if one model exhausts its retries, it automatically rotates to the next model in the registry, all capped by a total timeout budget so latency stays bounded. Long-term memory is also self-hosted—mem0 runs in-process and persists embeddings into your existing PostgreSQL via pgvector, meaning no separate memory vendor or cloud account is required.

Key highlights

• Circular model fallback with exponential backoff and a bounded total timeout budget
• Self-hosted long-term memory (mem0 + pgvector) per user, backed by an optional Valkey/Redis cache layer
• Langfuse tracing on all LLM calls, plus Prometheus metrics and Grafana dashboards
• JWT session auth and slowapi rate limiting included out of the box
• Structured logging that carries request, session, and user context on every line

Caveats

• LLM support is currently OpenAI only; multi-provider support (Anthropic, Google, OpenRouter) is planned but not yet implemented
• mem0 still requires a valid OPENAI_API_KEY for fact extraction and embeddings, even though the vector storage is self-hosted
• PostgreSQL with the pgvector extension is required for long-term memory and general persistence

Verdict

Reach for this if you have agent logic ready but need a production scaffold with auth, memory, and observability already wired together. Look elsewhere if you need a multi-provider LLM abstraction or want to avoid running PostgreSQL.