AWS-native ML infrastructure that outlived its maintainers

What it does

Cortex is an AWS-centric control plane that sits on top of EKS to run machine learning inference at scale. It abstracts cluster management by offering three workload modes—real-time, asynchronous, and batch—that autoscale based on request volume, queue depth, or job demand. The project handles GPU and CPU autoscaling, spot-instance scheduling with on-demand backups, and VPC isolation, essentially trying to turn raw AWS compute into a managed ML platform.

The interesting bit

The clever angle is grafting serverless semantics—request-based autoscaling, async queues, and on-demand batch jobs—onto traditional Kubernetes clusters. It also bakes in cost optimizations like spot instances with automated on-demand backups, which is exactly the kind of tedious infrastructure work most platform teams end up rebuilding from scratch.

Key highlights

• Supports real-time, async, and batch inference workloads with distinct autoscaling triggers
• Runs exclusively on AWS EKS with VPC isolation and IAM integration
• Includes a Terraform provider and declarative configuration for cluster provisioning
• Ships pre-built Grafana dashboards and CloudWatch log integrations
• Handles spot-instance scheduling with automatic on-demand fallback

Caveats

• No longer actively maintained by the original authors, so compatibility with future AWS APIs is uncertain
• AWS-only; there is no support for GCP, Azure, or on-premises clusters
• The README is vague on model packaging requirements and supported ML frameworks

Verdict

Study it if you are designing an internal ML platform and want a reference for EKS autoscaling and spot-instance orchestration. Avoid it if you need an actively maintained, multi-cloud inference layer for production workloads today.