What it does

This is a Unity-native implementation of pix2pix image-to-image translation — the “draw edges, get cats” neural network — that runs entirely inside the engine. It ships its own inference engine written in compute shaders, so you don’t need TensorFlow, PyTorch, or any external ML framework installed.

The interesting bit

The hard part isn’t the model; it’s the plumbing. Keijiro wrote a custom inference engine in C# and compute shaders rather than binding to ONNX or Barracade, which means the whole pipeline — load, inference, render — stays on the GPU without crossing back to CPU-land.

Key highlights

• Self-contained: no external ML frameworks, just Unity and a GPU with compute shader support (DX11/Metal/Vulkan)
• Targets the .pict weight format from the classic pix2pix-tensorflow models
• Includes a “Sketch Pad” demo: real-time edges-to-cats (or whatever you trained) with pre-built binaries available
• Optimized for NVIDIA architectures; the README suggests GTX 1070 or better for “flawless” demo performance
• Requires Unity 2018.1 or newer

Caveats

• No trained models included in the repo; you must download and drop in weights manually
• Only supports the older .pict format, not modern ONNX or SavedModel exports
• “Platform agnostic” in theory, but NVIDIA-leaning in practice; your mileage may vary on AMD/Apple Silicon

Verdict

Grab this if you’re building interactive Unity experiences and need pix2pix-style translation without dragging in a Python runtime. Skip it if you want plug-and-play models or need to deploy on low-end mobile GPUs.