Container freight for open-source language models

What it does AIKit is a Go-based platform that bundles open-source LLMs into container images for local inference, fine-tuning, and distribution. It uses LocalAI under the hood to provide an OpenAI-compatible REST API, so existing clients can talk to Llama, Mixtral, or Phi without code changes. The project also supports packaging models as OCI artifacts—complete with SBOMs and signed provenance—so they can be stored and shipped through any standard container registry.

The interesting bit The clever part isn’t just running models in containers; it’s treating the models themselves as container artifacts. AIKit supports the CNCF ModelPack specification, meaning a model image is just another piece of infrastructure you can version, scan, and pull from an air-gapped registry. That declarative approach extends to fine-tuning via Unsloth, letting you specify training jobs the same way you might specify a deployment manifest.

Key highlights

• Ships pre-made CPU and GPU images (CUDA and ROCm) for Llama 3, Mixtral, Phi 4, Gemma 2, and others—no GPU required for basic inference.
• OpenAI API compatibility means drop-in replacement for existing clients; multi-modal and image-generation models are also supported.
• Uses “chiseled” Ubuntu images to keep the attack surface small, and generates SBOMs plus signed provenance attestations for supply-chain hygiene.
• Runs fully offline in air-gapped environments using self-hosted or remote registries.
• Kubernetes-ready with declarative specs for both inference and fine-tuning workloads.

Verdict If you are a platform engineer or developer who wants to self-host open LLMs without managing Python environments or cloud API keys, AIKit is worth a look. If you need heavily customized model architectures or already have a mature ML pipeline, this is probably too opinionated.