Hand-rolling a deep learning inference engine in modern C++

What it does

KuiperInfer is a from-scratch C++17 inference framework built as a teaching tool. It implements its own tensor class, computation graph, and operators—convolution via im2col, pooling, activations, and an expression AST—then loads pretrained models in PNNX format to run ResNet, YOLOv5, U-Net, and MobileNet. The README is essentially a course syllabus: each section links to a Bilibili lecture, and the repo is described as the upstream pilot for that curriculum.

The interesting bit

Rather than wrapping an existing runtime, you hand-write the graph executor and memory layout logic yourself. The project uses Armadillo with OpenBLAS or Intel MKL for the heavy linear algebra, but the high-level graph scheduling and operator implementations are original code, giving you a ground-up view of how an inference engine actually works.

Key highlights

• Core operators implemented from scratch: Convolution, MaxPooling, BatchNorm, Softmax, SiLU, HardSwish, and an expression parser with AST
• Model ingestion via PNNX (from the ncnn ecosystem) rather than the more common ONNX route
• Working demos for U-Net semantic segmentation and YOLOv5 object detection using pretrained weights
• Uses OpenMP for parallelization and supports Google Test and Google Benchmark for verification
• Explicitly aimed at job-interview preparation and deep-learning systems education

Caveats

• The README is heavily interleaved with promotion for a separate, newer commercial course on LLM inference with CUDA; the open-source repo itself appears focused on classical CV models and CPU inference
• Some README details, like the cited star count, lag behind the repository’s actual metrics
• PNNX is less ubiquitous than ONNX, so bringing your own models may require extra conversion tooling not covered in the README

Verdict

Worth cloning if you are a C++ developer or student who wants to understand inference engines by building one, or if you need a substantive interview project. Look elsewhere if you need a drop-in, battle-tested replacement for ONNX Runtime or TensorRT.