← all repositories
amitshekhariitbhu/ai-system-design

From KV cache to A2A: one README maps the AI systems territory

A Markdown-only syllabus for the discipline that appeared overnight: wiring GPUs, caches, RAG, and agents into one system that is fast, cheap, reliable, and safe.

★1.1k stars Markdown LearningLLMOps · Eval
ai-system-design
Velocity · 7d
+59
★ / day
Trend
↗accelerating
not enough history yet
star history

What it does This is a reading repository, not software — no code, nothing to run. It’s a long, ordered guide to AI system design: how an LLM actually runs on a GPU, how prefill and decode shape latency, how caching, routing, and batching cut cost, and how RAG, agents, guardrails, and evals assemble into a whole system. It closes with an eight-step framework for attacking any AI system design problem, plus interview questions, a glossary, and FAQs.

The interesting bit The unglamorous plumbing gets equal billing with the fashionable parts. Load balancing by least outstanding tokens, prefix-aware routing, prefill-decode disaggregation, KV cache compression — the material most LLM tutorials skip because it isn’t a prompt trick. The guide also explicitly frames AI system design as distinct from classic system design, which is the right call: tokens, TTFT, and GPU memory are constraints the CRUD world never had.

Key highlights

  • Full serving-stack coverage: TTFT, tokens per second, chunked prefill, speculative decoding, GQA, MoE, quantization
  • RAG past the basics: hybrid search, HyDE, ColBERT late interaction, GraphRAG, vectorless RAG
  • Agent plumbing: MCP (including when it’s overkill), the A2A protocol, agent memory stacks, multi-agent coordination patterns
  • Case studies on how Claude Code and Cursor work, plus designing a real-time voice AI agent
  • Cost optimization as its own section with seven concrete levers, not an afterthought

Caveats

  • The repo is the map, not the territory: the “detailed blogs for every deep dive” live outside the repo, so real depth means following links
  • Self-declared work in progress — the author states it will keep growing as new topics land
  • It doubles as Outcome School content marketing; expect course signposts between sections

Verdict Worth a bookmark if you’re a backend engineer moving into LLM products and want the whole landscape indexed before diving in — or if an AI system design interview is in your future. If you wanted a tool or library, this isn’t that; it’s a syllabus.

Frequently asked

What is amitshekhariitbhu/ai-system-design?
A Markdown-only syllabus for the discipline that appeared overnight: wiring GPUs, caches, RAG, and agents into one system that is fast, cheap, reliable, and safe.
Is ai-system-design open source?
Yes — amitshekhariitbhu/ai-system-design is open source, released under the Apache-2.0 license.
What language is ai-system-design written in?
amitshekhariitbhu/ai-system-design is primarily written in Markdown.
How popular is ai-system-design?
amitshekhariitbhu/ai-system-design has 1.1k stars on GitHub and is currently accelerating.
Where can I find ai-system-design?
amitshekhariitbhu/ai-system-design is on GitHub at https://github.com/amitshekhariitbhu/ai-system-design.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.