Taming model sprawl with Kafka and overcommit

What it does

Seldon Core 2 is a Kubernetes control plane for running machine learning systems at scale. It wraps models, custom logic, and drift detectors into composable pipelines that stream data over Kafka rather than simple request/response loops. The framework handles autoscaling, packs multiple models onto shared inference servers, and will even overcommit memory—letting you deploy more models than your cluster technically has RAM for, on the assumption that most are cold at any given moment.

The interesting bit

By treating an AI application as a data-flow graph instead of a static endpoint, Seldon Core 2 makes shadow deployments and A/B tests a matter of routing configuration. The bet on Kafka-backed pipelines between components is the kind of architectural choice that either absorbs your scaling headaches or introduces operational complexity you did not know you had.

Key highlights

• Composable pipelines connect models and custom components via Kafka real-time streams
• Multi-model serving consolidates many models onto shared inference servers to reduce infrastructure costs
• Memory overcommit allows deploying more models than available RAM, banking on usage sparsity
• Native experiment routing supports A/B tests and shadow deployments across candidate pipelines
• Extensible with custom components for drift detection, outlier detection, and LLM integration

Caveats

• Distributed under the Business Source License, which is not a traditional open-source license; read the LICENSE file before embedding it in commercial workflows
• The README is heavy on feature bullets and light on architecture detail, so you will likely need the external docs to understand how custom resources and Kafka topics map to your cluster
• Positioned within a broader Seldon commercial ecosystem, with an explicit nudge to contact Seldon for commercial use

Verdict Evaluate it if your Kubernetes estate is turning into a model zoo that needs pipeline routing and traffic splitting. Look elsewhere if you want a lightweight inference server or a project under a permissive open-source license.