CPU face tracking built for puppeteering, not selfies

What it does

OpenSeeFace is a real-time facial landmark tracker that runs entirely on the CPU. It detects face pose and expression landmarks, then ships that data over UDP to a separate receiver—most commonly a Unity component, though it also powers VTube Studio and Godot projects. It is explicitly a tracking library, not a finished avatar application, so it must be wired into another application.

The interesting bit

The landmarks are deliberately tweaked for animating avatars rather than matching facial geometry pixel-for-pixel. The model uses quasi-3D face contours and sacrifices some eye-region accuracy in exchange for stability across extreme head poses, bad lighting, and low-resolution webcams. There is even a “toaster” model tier that hits 213 fps if you are willing to trade quality for frame rate.

Key highlights

• Five speed/quality tiers (model -1 through 3), ranging from 213 fps to 44 fps on a single CPU core
• MobileNetV3 backbone converted to ONNX for inference via ONNX Runtime
• UDP streaming lets the tracker run on a separate machine from the renderer, keeping raw webcam footage off the animation PC
• Unity components included for receiving landmarks, visualizing points, and launching the tracker process
• Robust to partial occlusion, glasses, and noisy low-light footage; expression detection is available but requires per-user calibration

Caveats

• Eye-region tracking is less accurate than MediaPipe, according to the author’s own comparison
• Expression detection requires manual calibration for each user; there is no one-size-fits-all model
• Not a standalone program: it is command-line driven and leaves UDP integration to the user

Verdict

A solid choice for VTubers, indie avatar apps, or anyone who needs offline, CPU-only face tracking they can embed in a pipeline. Skip it if you want a polished out-of-the-box puppeteering app or research-grade facial reconstruction.