From KV cache to A2A: one README maps the AI systems territory
A Markdown-only syllabus for the discipline that appeared overnight: wiring GPUs, caches, RAG, and agents into one system that is fast, cheap, reliable, and safe.

What it does This is a reading repository, not software — no code, nothing to run. It’s a long, ordered guide to AI system design: how an LLM actually runs on a GPU, how prefill and decode shape latency, how caching, routing, and batching cut cost, and how RAG, agents, guardrails, and evals assemble into a whole system. It closes with an eight-step framework for attacking any AI system design problem, plus interview questions, a glossary, and FAQs.
The interesting bit The unglamorous plumbing gets equal billing with the fashionable parts. Load balancing by least outstanding tokens, prefix-aware routing, prefill-decode disaggregation, KV cache compression — the material most LLM tutorials skip because it isn’t a prompt trick. The guide also explicitly frames AI system design as distinct from classic system design, which is the right call: tokens, TTFT, and GPU memory are constraints the CRUD world never had.
Key highlights
- Full serving-stack coverage: TTFT, tokens per second, chunked prefill, speculative decoding, GQA, MoE, quantization
- RAG past the basics: hybrid search, HyDE, ColBERT late interaction, GraphRAG, vectorless RAG
- Agent plumbing: MCP (including when it’s overkill), the A2A protocol, agent memory stacks, multi-agent coordination patterns
- Case studies on how Claude Code and Cursor work, plus designing a real-time voice AI agent
- Cost optimization as its own section with seven concrete levers, not an afterthought
Caveats
- The repo is the map, not the territory: the “detailed blogs for every deep dive” live outside the repo, so real depth means following links
- Self-declared work in progress — the author states it will keep growing as new topics land
- It doubles as Outcome School content marketing; expect course signposts between sections
Verdict Worth a bookmark if you’re a backend engineer moving into LLM products and want the whole landscape indexed before diving in — or if an AI system design interview is in your future. If you wanted a tool or library, this isn’t that; it’s a syllabus.
Frequently asked
- What is amitshekhariitbhu/ai-system-design?
- A Markdown-only syllabus for the discipline that appeared overnight: wiring GPUs, caches, RAG, and agents into one system that is fast, cheap, reliable, and safe.
- Is ai-system-design open source?
- Yes — amitshekhariitbhu/ai-system-design is open source, released under the Apache-2.0 license.
- What language is ai-system-design written in?
- amitshekhariitbhu/ai-system-design is primarily written in Markdown.
- How popular is ai-system-design?
- amitshekhariitbhu/ai-system-design has 1.1k stars on GitHub and is currently accelerating.
- Where can I find ai-system-design?
- amitshekhariitbhu/ai-system-design is on GitHub at https://github.com/amitshekhariitbhu/ai-system-design.